Optimizations for filter-based join algorithms in MapReduce

被引:2
|
作者
Rababa, Salahaldeen [1 ]
Al-Badarneh, Amer [2 ]
机构
[1] Jordan Univ Sci & Technol, Comp Engn Dept, Irbid, Jordan
[2] Jordan Univ Sci & Technol, Comp Informat Syst Dept, Irbid, Jordan
关键词
Join algorithms; big data management; query optimization; MapReduce; DISTRIBUTED JOINS;
D O I
10.3233/JIFS-201220
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Large-scale datasets collected from heterogeneous sources often require a join operation to extract valuable information. MapReduce is an efficient programming model for processing large-scale data. However, it has some limitations in processing heterogeneous datasets. This is because of the large amount of redundant intermediate records that are transferred through the network. Several filtering techniques have been developed to improve the join performance, but they require multiple MapReduce jobs to process the input datasets. To address this issue, the adaptive filter-based join algorithms are presented in this paper. Specifically, three join algorithms are introduced to perform the processes of filters creation and redundant records elimination within a single MapReduce job. A cost analysis of the introduced join algorithms shows that the I/O cost is reduced compared to the state-of-the-art filter-based join algorithms. The performance of the join algorithms was evaluated in terms of the total execution time and the total amount of I/O data transferred. The experimental results show that the adaptive Bloom join, semi-adaptive intersection Bloom join, and adaptive intersection Bloom join decrease the total execution time by 30%, 25%, and 35%, respectively; and reduce the total amount of I/O data transferred by 18%, 25%, and 50%, respectively.
引用
收藏
页码:8963 / 8980
页数:18
相关论文
共 50 条
  • [31] An Empirical Study of Filter-based Feature Selection Algorithms Using Noisy Training Data
    Yuan, Weiwei
    Guan, Donghai
    Shen, Linshan
    Pan, Haiwei
    2014 4TH IEEE INTERNATIONAL CONFERENCE ON INFORMATION SCIENCE AND TECHNOLOGY (ICIST), 2014, : 209 - 212
  • [32] An experiment comparing double exponential smoothing and Kalman filter-based predictive tracking algorithms
    LaViola, JJ
    IEEE VIRTUAL REALITY 2003, PROCEEDINGS, 2003, : 283 - 284
  • [33] A comprehensive benchmark between two filter-based multiple-point simulation algorithms
    Savadkoohi, M. Bavand
    Tokhmechi, B.
    Gloaguen, E.
    Arab-Amiri, A. R.
    JOURNAL OF MINING AND ENVIRONMENT, 2019, 10 (01): : 139 - 149
  • [34] MapReduce: Limitations, Optimizations and Open Issues
    Kalavri, Vasiliki
    Vlassov, Vladimir
    2013 12TH IEEE INTERNATIONAL CONFERENCE ON TRUST, SECURITY AND PRIVACY IN COMPUTING AND COMMUNICATIONS (TRUSTCOM 2013), 2013, : 1031 - 1038
  • [35] Selection of Optimal Object Features in Object-Based Image Analysis Using Filter-Based Algorithms
    Ismail Colkesen
    Taskin Kavzoglu
    Journal of the Indian Society of Remote Sensing, 2018, 46 : 1233 - 1242
  • [36] Selection of Optimal Object Features in Object-Based Image Analysis Using Filter-Based Algorithms
    Colkesen, Ismail
    Kavzoglu, Taskin
    JOURNAL OF THE INDIAN SOCIETY OF REMOTE SENSING, 2018, 46 (08) : 1233 - 1242
  • [37] A Filter-Based Controller for a Buck Converter
    Mohebbi, Mohammad
    McIntyre, Michael L.
    Latham, Joseph
    Rivera, Pablo
    2017 IEEE 18TH WORKSHOP ON CONTROL AND MODELING FOR POWER ELECTRONICS (COMPEL), 2017,
  • [38] Filter-Based Fading Channel Modeling
    Alimohammad, Amirhossein
    Fard, Saeed Fouladi
    Cockburn, Bruce F.
    MODELLING AND SIMULATION IN ENGINEERING, 2012, 2012
  • [39] Fast Filter-Based Boolean Matchers
    Yu, Chaofan
    Wang, Lingli
    Zhang, Chun
    Hu, Yu
    He, Lei
    IEEE EMBEDDED SYSTEMS LETTERS, 2013, 5 (04) : 65 - 68
  • [40] Optimal filter-based detection of microcalcifications
    Gulsrud, TO
    Husoy, JH
    IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, 2001, 48 (11) : 1272 - 1281