Optimizations for filter-based join algorithms in MapReduce

被引:2
|
作者
Rababa, Salahaldeen [1 ]
Al-Badarneh, Amer [2 ]
机构
[1] Jordan Univ Sci & Technol, Comp Engn Dept, Irbid, Jordan
[2] Jordan Univ Sci & Technol, Comp Informat Syst Dept, Irbid, Jordan
关键词
Join algorithms; big data management; query optimization; MapReduce; DISTRIBUTED JOINS;
D O I
10.3233/JIFS-201220
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Large-scale datasets collected from heterogeneous sources often require a join operation to extract valuable information. MapReduce is an efficient programming model for processing large-scale data. However, it has some limitations in processing heterogeneous datasets. This is because of the large amount of redundant intermediate records that are transferred through the network. Several filtering techniques have been developed to improve the join performance, but they require multiple MapReduce jobs to process the input datasets. To address this issue, the adaptive filter-based join algorithms are presented in this paper. Specifically, three join algorithms are introduced to perform the processes of filters creation and redundant records elimination within a single MapReduce job. A cost analysis of the introduced join algorithms shows that the I/O cost is reduced compared to the state-of-the-art filter-based join algorithms. The performance of the join algorithms was evaluated in terms of the total execution time and the total amount of I/O data transferred. The experimental results show that the adaptive Bloom join, semi-adaptive intersection Bloom join, and adaptive intersection Bloom join decrease the total execution time by 30%, 25%, and 35%, respectively; and reduce the total amount of I/O data transferred by 18%, 25%, and 50%, respectively.
引用
收藏
页码:8963 / 8980
页数:18
相关论文
共 50 条
  • [41] Discrete FIR filter-based Control
    Cortes-Romero, John
    Gomez-Leon, Brian
    Sira-Ramirez, Hebertt
    ISA TRANSACTIONS, 2025, 157 : 591 - 602
  • [42] Washout Filter-Based Power Sharing
    Yazdanian, Mehrdad
    Mehrizi-Sani, Ali
    IEEE TRANSACTIONS ON SMART GRID, 2016, 7 (02) : 967 - 968
  • [43] A Filter-Based Format Conversion Approach
    Jeon, Gwanggil
    Kang, SeokHoon
    Lee, Young-Sup
    CONVERGENCE AND HYBRID INFORMATION TECHNOLOGY, 2012, 310 : 559 - 565
  • [44] Filter-Based Wilkinson Power Divider
    Chau, Wei-Ming
    Hsu, Ko-Wen
    Tu, Wen-Hua
    IEEE MICROWAVE AND WIRELESS COMPONENTS LETTERS, 2014, 24 (04) : 239 - 241
  • [45] Filter-based unsteady RANS computations
    Johansen, ST
    Wu, JY
    Shyy, W
    INTERNATIONAL JOURNAL OF HEAT AND FLUID FLOW, 2004, 25 (01) : 10 - 21
  • [46] Orthogonal Filter-Based Networks for Learning
    Sienko, Wieslaw
    Citko, Wieslaw
    ADVANCES IN COGNITIVE NEURODYNAMICS, PROCEEDINGS, 2008, : 873 - +
  • [47] Selection of Relevant Geometric Features Using Filter-Based Algorithms for Point Cloud Semantic Segmentation
    Atik, Muhammed Enes
    Duran, Zaide
    ELECTRONICS, 2022, 11 (20)
  • [48] Kalman filter-based algorithms for monitoring the ionosphere and plasmasphere with GPS in near-real time
    Anghel, Adela
    Carrano, Charles
    Komjathy, Attila
    Astilean, Adina
    Letia, Tiberiu
    JOURNAL OF ATMOSPHERIC AND SOLAR-TERRESTRIAL PHYSICS, 2009, 71 (01) : 158 - 174
  • [49] Bloom filter and its variants for the optimization of MapReduce's algorithms: A review
    Ezzaki, F.
    Abghour, N.
    Elomri, A.
    Moussaid, K.
    Rida, M.
    PROCEEDINGS OF 2020 5TH INTERNATIONAL CONFERENCE ON CLOUD COMPUTING AND ARTIFICIAL INTELLIGENCE: TECHNOLOGIES AND APPLICATIONS (CLOUDTECH'20), 2020, : 175 - 181
  • [50] Scaling Bloom filter-based multicast via filter switching
    Tsilopoulos, Christos
    Xylomenos, George
    2013 IEEE SYMPOSIUM ON COMPUTERS AND COMMUNICATIONS (ISCC), 2013,