Design and Implementation of Scheduling Pool Scheduling Algorithm Based on Reuse of Jobs in Spark

被引：3
|
作者：
Tang Jianchao [1 ]
Yang Shuqiang [1 ]
Huang Chaoqiang [1 ]
Yan Zhou [1 ]
机构：
[1] Natl Univ Def Technol, Coll Comp Sci, Changsha, Hunan, Peoples R China
来源：
2016 IEEE FIRST INTERNATIONAL CONFERENCE ON DATA SCIENCE IN CYBERSPACE (DSC 2016) | 2016年
基金：
中国国家自然科学基金;
关键词：
scheduling pool; reuse of jobs; Spark; scheduling algorithm;
D O I：
10.1109/DSC.2016.81
中图分类号：
TP [自动化技术、计算机技术];
学科分类号：
0812 ;
摘要：
As a distributed computing framework based on memory, Spark is being used by more and more enterprises. Generally, Spark runs in multi-user and multi-job mode, where may exist a large number of reuse of jobs. This reuse, here, refers to the calculation reuse inside the jobs, and it can greatly shorten the executing time of jobs in Spark. Therefore, this paper proposes a scheduling pool scheduling algorithm based on reuse of jobs. This algorithm is based on the original scheduling pool scheduling algorithm in Spark and can take great advantage of the reusable parts. Experiments show that the new scheduling algorithm realizes reuse of jobs, and improves the execution efficiency of the cluster.
引用
收藏
页码：290 / 295
页数：6
相关论文
共 50 条
  • [1] Adaptive Scheduling of Parallel Jobs in Spark Streaming
    Cheng, Dazhao
    Chen, Yuan
    Zhou, Xiaobo
    Gmach, Daniel
    Milojicic, Dejan
    IEEE INFOCOM 2017 - IEEE CONFERENCE ON COMPUTER COMMUNICATIONS, 2017,
  • [2] Optimal Reuse Design Scheduling of Mine Water Based on Improved Whale Algorithm
    Yue, Yuangan
    Liu, Yang
    Bo, Lei
    Zhang, Zihang
    Yang, Hongwei
    Wang, Yiying
    SENSORS, 2022, 22 (14)
  • [3] An algorithm for scheduling jobs in hypercube systems
    Kwon, OH
    Chwa, KY
    IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, 1998, 9 (09) : 856 - 860
  • [4] An effective algorithm of jobs scheduling in clusters
    Yi, Shengwei
    Wang, Zhichao
    Ma, Shilong
    Che, Zhanbin
    Huang, Yonggang
    Chen, Xin
    Journal of Computational Information Systems, 2010, 6 (10): : 3163 - 3172
  • [5] A delay scheduling algorithm for short jobs
    Liu, Qiang
    Dong, Xiaoshe
    Zhu, Zhengdong
    Wang, Yinfeng
    Hsi-An Chiao Tung Ta Hsueh/Journal of Xi'an Jiaotong University, 2015, 49 (02): : 1 - 5
  • [6] SLA-Based Scheduling of Spark Jobs in Hybrid Cloud Computing Environments
    Islam, Muhammed Tawfiqul
    Wu, Huaming
    Karunasekera, Shanika
    Buyya, Rajkumar
    IEEE TRANSACTIONS ON COMPUTERS, 2022, 71 (05) : 1117 - 1132
  • [7] BSM:A scheduling algorithm for dynamic jobs based on economics theory
    Cao, Bo
    Wu, Yongwei
    Yang, Guangwen
    Liu, Jia
    Jiang, Jianjin
    GCC 2005: FIFTH INTERNATIONAL CONFERENCE ON GRID AND COOPERATIVE COMPUTING, PROCEEDINGS, 2006, : 62 - +
  • [8] The list scheduling algorithm for scheduling unreliable jobs on two parallel machines
    Agnetis, Alessandro
    Detti, Paolo
    Pranzo, Marco
    DISCRETE APPLIED MATHEMATICS, 2014, 165 : 2 - 11
  • [9] A List Scheduling Algorithm for Scheduling Multi-user Jobs on Clusters
    Barbosa, J.
    Monteiro, A. P.
    HIGH PERFORMANCE COMPUTING FOR COMPUTATIONAL SCIENCE - VECPAR 2008, 2008, 5336 : 123 - +
  • [10] The Design and Implementation of an Optimized MPTCP Data Scheduling Algorithm
    Li, Ximing
    Wang, Wendong
    COMPUTER AND INFORMATION TECHNOLOGY, 2014, 519-520 : 170 - 176
12345