On the role of message broker middleware for many-task computing on a big-data platform

被引：4
|
作者：
Cao Ngoc Nguyen
Jaehwan Lee
Soonwook Hwang
Jik-Soo Kim
机构：
[1] University of Science & Technology,Korea Institute of Science and Technology Information
[2] Korea Aerospace University,School of Electronics and Information Engineering
[3] Myongji University,Department of Computer Engineering
来源：
Cluster Computing | 2019年 / 22卷
关键词：
Many-task computing; Message broker middleware; Hadoop; YARN; ActiveMQ; Kafka; MOHA; Load balancing;
D O I：
暂无
中图分类号：
学科分类号：
摘要：
We have designed and implemented a new data processing framework called “Many-task computing On HAdoop” (MOHA) which aims to effectively support fine-grained many-task applications that can show another type of data-intensive workloads in the YARN-based Hadoop 2.0 platform. MOHA is developed as one of Hadoop YARN applications so that it can transparently co-host existing many-task computing (MTC) applications with other data processing workflows such as MapReduce in a single Hadoop cluster. In this paper, we investigate main characteristics of two well-known open-source message broker middleware systems (Apache ActiveMQ and Kafka) and their implications on a many-task management scheme in our MOHA framework. Through our extensive experiments with a real MTC application, we demonstrate and discuss trade-offs between parallelism and load balancing of data access patterns in message broker middleware systems for Many-Task Computing on Hadoop.
引用
收藏
页码：2527 / 2540
页数：13
相关论文
共 50 条
  • [21] MTCProv: a practical provenance query framework for many-task scientific computing
    Gadelha, Luiz M. R., Jr.
    Wilde, Michael
    Mattoso, Marta
    Foster, Ian
    DISTRIBUTED AND PARALLEL DATABASES, 2012, 30 (5-6) : 351 - 370
  • [22] Low Power and Scalable Many-Core Architecture for Big-Data Stream Computing
    Kanoun, Karim
    Ruggiero, Martino
    Atienza, David
    van der Schaar, Mihaela
    2014 IEEE COMPUTER SOCIETY ANNUAL SYMPOSIUM ON VLSI (ISVLSI), 2014, : 469 - 474
  • [23] Efficient algorithms for frequent pattern mining in many-task computing environments
    Lin, Kawuu W.
    Lo, Yu-Chin
    KNOWLEDGE-BASED SYSTEMS, 2013, 49 : 10 - 21
  • [24] Enabling Docker Containers for High-Performance and Many-Task Computing
    Azab, Abdulrahman
    2017 IEEE INTERNATIONAL CONFERENCE ON CLOUD ENGINEERING (IC2E 2017), 2017, : 279 - 285
  • [25] The Scaling of Many-Task Computing Approaches in Python']Python on Cluster Supercomputers
    Lunacek, Monte
    Braden, Jazcek
    Hauser, Thomas
    2013 IEEE INTERNATIONAL CONFERENCE ON CLUSTER COMPUTING (CLUSTER), 2013,
  • [26] Developing political-ecological theory: The need for many-task computing
    Haas, Timothy
    PLOS ONE, 2020, 15 (11):
  • [27] Emotion-based Social Computing Platform for Streaming Big-data: Architecture and Application
    Zhang, Leihan
    Zhao, Jichang
    Xu, Ke
    2016 13TH INTERNATIONAL CONFERENCE ON SERVICE SYSTEMS AND SERVICE MANAGEMENT, 2016,
  • [28] Multi-core versus many-core computing for many-task Branch-and-Bound applied to big optimization problems
    Melab, N.
    Gmys, J.
    Mezmaz, M.
    Tuyttens, D.
    FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE, 2018, 82 : 472 - 481
  • [29] Big-data in cloud computing: a taxonomy of risks
    Miller, Holmes E.
    INFORMATION RESEARCH-AN INTERNATIONAL ELECTRONIC JOURNAL, 2013, 18 (01):
  • [30] Efficient Strategies for Many-task Frequent Pattern Mining in Cloud Computing Environments
    Lin, Kawuu W.
    Luo, Yu-Chin
    2010 IEEE INTERNATIONAL CONFERENCE ON SYSTEMS, MAN AND CYBERNETICS (SMC 2010), 2010,
12345