MapReduce Job Scheduling Based on Remaining Job Sizes

被引:0
|
作者
Matsuki, Tatsuma [1 ]
Takine, Tetsuya [1 ]
机构
[1] Osaka Univ, Grad Sch Engn, Suita, Osaka 5650871, Japan
关键词
MapReduce; Hadoop; job scheduling;
D O I
10.1587/transcom.E98.B.180
中图分类号
TM [电工技术]; TN [电子技术、通信技术];
学科分类号
0808 ; 0809 ;
摘要
The MapReduce job scheduler implemented in Hadoop is a mechanism to decide which job is allowed to use idle resources in Hadoop. In terms of the mean job response time, the performance of the job scheduler strongly depends on the job arrival pattern, which includes job size (i.e., the amount of required resources) and their arrival order. Because existing schedulers do not utilize information about job sizes, however, those schedulers suffer severe performance degradation with some arrival patterns. In this paper, we propose a scheduler that estimates and utilizes remaining job sizes, in order to achieve good performance regardless of job arrival patterns. Through simulation experiments, we confirm that for various arrival patterns, the proposed scheduler achieves better performance than the existing schedulers.
引用
收藏
页码:180 / 189
页数:10
相关论文
共 50 条
  • [1] A review on job scheduling for hadoop mapreduce
    Kalia, Khushboo
    Gupta, Neeraj
    Proceedings - 2017 International Conference on Next Generation Computing and Information Systems, ICNGCIS 2017, 2018, : 86 - 91
  • [2] A COMPARATIVE REVIEW OF JOB SCHEDULING FOR MAPREDUCE
    Yoo, Dongjin
    Sim, Kwang Mong
    2011 IEEE INTERNATIONAL CONFERENCE ON CLOUD COMPUTING AND INTELLIGENCE SYSTEMS, 2011, : 353 - 358
  • [3] A REVIEW ON JOB SCHEDULING FOR HADOOP MAPREDUCE
    Kalia, Khushboo
    Gupta, Neeraj
    2017 INTERNATIONAL CONFERENCE ON NEXT GENERATION COMPUTING AND INFORMATION SYSTEMS (ICNGCIS), 2017, : 75 - 79
  • [4] Job Shop Scheduling Problem with Job Sizes and Inventories
    Shen Xinyi
    Wang Aimin
    Yan, Ge
    Ye Jieran
    PROCEEDINGS OF 2020 IEEE 11TH INTERNATIONAL CONFERENCE ON MECHANICAL AND INTELLIGENT MANUFACTURING TECHNOLOGIES (ICMIMT 2020), 2020, : 202 - 206
  • [5] A Cross-job Framework for MapReduce Scheduling
    Xiao, Xuejie
    Tang, Jian
    Chen, Zhenhua
    Xu, Jielong
    Wang, Chonggang
    2014 IEEE INTERNATIONAL CONFERENCE ON BIG DATA (BIG DATA), 2014, : 135 - 140
  • [6] Analysis of Job Scheduling Algorithms and Studying Dynamic Job Ordering to Optimize MapReduce
    Mohammed, Ahmed Qasim
    Bharati, Rajesh
    INTERNATIONAL CONFERENCE ON INTELLIGENT COMPUTING AND APPLICATIONS, ICICA 2016, 2018, 632 : 343 - 352
  • [7] Minimizing total job completion time in MapReduce scheduling
    Dong, Jianming
    Goebel, Randy
    Hu, Jueliang
    Lin, Guohui
    Su, Bing
    COMPUTERS & INDUSTRIAL ENGINEERING, 2021, 158
  • [8] Hadoop-MapReduce Job Scheduling Algorithms Survey
    Mohamed, Ehab
    Hong, Zheng
    2016 7TH INTERNATIONAL CONFERENCE ON CLOUD COMPUTING AND BIG DATA (CCBD), 2016, : 237 - 242
  • [9] Minimizing total job completion time in MapReduce scheduling
    Dong, Jianming
    Goebel, Randy
    Hu, Jueliang
    Lin, Guohui
    Su, Bing
    Computers and Industrial Engineering, 2021, 158
  • [10] An Improved Job Scheduling Algorithm by Utilizing Released Resources for MapReduce
    Garai, Chandan
    Dasgupta, Ranjan
    2014 FOURTH INTERNATIONAL CONFERENCE OF EMERGING APPLICATIONS OF INFORMATION TECHNOLOGY (EAIT), 2014, : 9 - 14