Optimizations for filter-based join algorithms in MapReduce

被引:2
|
作者
Rababa, Salahaldeen [1 ]
Al-Badarneh, Amer [2 ]
机构
[1] Jordan Univ Sci & Technol, Comp Engn Dept, Irbid, Jordan
[2] Jordan Univ Sci & Technol, Comp Informat Syst Dept, Irbid, Jordan
关键词
Join algorithms; big data management; query optimization; MapReduce; DISTRIBUTED JOINS;
D O I
10.3233/JIFS-201220
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Large-scale datasets collected from heterogeneous sources often require a join operation to extract valuable information. MapReduce is an efficient programming model for processing large-scale data. However, it has some limitations in processing heterogeneous datasets. This is because of the large amount of redundant intermediate records that are transferred through the network. Several filtering techniques have been developed to improve the join performance, but they require multiple MapReduce jobs to process the input datasets. To address this issue, the adaptive filter-based join algorithms are presented in this paper. Specifically, three join algorithms are introduced to perform the processes of filters creation and redundant records elimination within a single MapReduce job. A cost analysis of the introduced join algorithms shows that the I/O cost is reduced compared to the state-of-the-art filter-based join algorithms. The performance of the join algorithms was evaluated in terms of the total execution time and the total amount of I/O data transferred. The experimental results show that the adaptive Bloom join, semi-adaptive intersection Bloom join, and adaptive intersection Bloom join decrease the total execution time by 30%, 25%, and 35%, respectively; and reduce the total amount of I/O data transferred by 18%, 25%, and 50%, respectively.
引用
收藏
页码:8963 / 8980
页数:18
相关论文
共 50 条
  • [2] Correlation Filter-based Object Tracking Algorithms
    Zhao, Songke
    Sun, Kewei
    Ji, Yuanfa
    Guo, Ning
    Jia, Xizi
    2020 IEEE 3RD INTERNATIONAL CONFERENCE ON INFORMATION COMMUNICATION AND SIGNAL PROCESSING (ICICSP 2020), 2020, : 57 - 62
  • [3] Cloud MapReduce for Particle Filter-based Data Assimilation for Wildfire Spread Simulation
    Bai, Fan
    Hu, Xiaolin
    HIGH PERFORMANCE COMPUTING SYMPOSIUM 2013 (HPC 2013) - 2013 SPRING SIMULATION MULTI-CONFERENCE (SPRINGSIM'13), 2013, 45 (06): : 91 - 96
  • [4] A Bloom Filter-based Approach for Efficient MapReduce Query Processing on Ordered Datasets
    Chen, Zhijian
    Wu, Dan
    Xie, Wenyan
    Zeng, Jiazhi
    He, Jian
    Wu, Di
    2013 INTERNATIONAL CONFERENCE ON ADVANCED CLOUD AND BIG DATA (CBD), 2013, : 93 - 98
  • [5] Handling data skew in join algorithms using MapReduce
    Myung, Jaeseok
    Shim, Junho
    Yeon, Jongheum
    Lee, Sang-goo
    EXPERT SYSTEMS WITH APPLICATIONS, 2016, 51 : 286 - 299
  • [6] A Novel KNN Join Algorithms based on Hilbert R-tree in MapReduce
    Du, Qinsheng
    Li, Xiongfei
    2013 3RD INTERNATIONAL CONFERENCE ON COMPUTER SCIENCE AND NETWORK TECHNOLOGY (ICCSNT), 2013, : 417 - 420
  • [7] A Survey on Parallel Join Algorithms Using MapReduce on Hadoop
    Barhoush, Malek Mahmoud
    AlSobeh, Anas Mohammad
    Al Rawashdeh, Ahmad
    2019 IEEE JORDAN INTERNATIONAL JOINT CONFERENCE ON ELECTRICAL ENGINEERING AND INFORMATION TECHNOLOGY (JEEIT), 2019, : 381 - 388
  • [8] Iterative and sequential Kalman filter-based speech enhancement algorithms
    Gannot, S
    Burshtein, D
    Weinstein, E
    IEEE TRANSACTIONS ON SPEECH AND AUDIO PROCESSING, 1998, 6 (04): : 373 - 385
  • [9] Load balancing in join algorithms for skewed data in MapReduce systems
    Elaheh Gavagsaz
    Ali Rezaee
    Hamid Haj Seyyed Javadi
    The Journal of Supercomputing, 2019, 75 : 228 - 254
  • [10] Improvement of Join Algorithms for Low-Selectivity Joins on MapReduce
    Matono, Akiyoshi
    Ogawa, Hirotaka
    Kojima, Isao
    DATABASES THEORY AND APPLICATIONS, 2015, 9093 : 117 - 128