IMapC: Inner MAPping Combiner to Enhance the Performance of MapReduce in Hadoop

被引：4

作者：

Kavitha, C. ^{[1
]}

Srividhya, S. R. ^{[1
]}

Lai, Wen-Cheng ^{[2
,3
]}

Mani, Vinodhini ^{[1
]}

机构：

[1] Sathyabama Inst Sci & Technol, Dept Comp Sci & Engn, Chennai 600119, Tamil Nadu, India

[2] Natl Yunlin Univ Sci & Technol, Bachelor Program Ind Projects, Touliu 640301, Yunlin, Taiwan

[3] Natl Yunlin Univ Sci & Technol, Dept Elect Engn, Touliu 640301, Yunlin, Taiwan

来源：

ELECTRONICS | 2022年 / 11卷 / 10期

关键词：

big data; combiner; distributed storage; hadoop; mapreduce; sort; task failure resilience; wordcount;

D O I：

10.3390/electronics11101599

中图分类号：

TP [自动化技术、计算机技术];

学科分类号：

0812 ;

摘要：

Hadoop is a framework for storing and processing huge amounts of data. With HDFS, large data sets can be managed on commodity hardware. MapReduce is a programming model for processing vast amounts of data in parallel. Mapping and reducing can be performed by using the MapReduce programming framework. A very large amount of data is transferred from Mapper to Reducer without any filtering or recursion, resulting in overdrawn bandwidth. In this paper, we introduce an algorithm called Inner MAPping Combiner (IMapC) for the map phase. This algorithm in the Mapper combines the values of recurring keys. In order to test the efficiency of the algorithm, different approaches were tested. According to the test, MapReduce programs that are implemented with the Default Combiner (DC) of IMapC will be 70% more efficient than those that are implemented without one. To make computations significantly faster, this work can be combined with MapReduce.

引用

页数：16

共 50 条

[1] Join Operations to Enhance Performance in Hadoop MapReduce Environment
Pagadala, Pavan Kumar
Vikram, M.
Eswarawaka, Rajesh
Reddy, P. Srinivasa
PROCEEDINGS OF THE 5TH INTERNATIONAL CONFERENCE ON FRONTIERS IN INTELLIGENT COMPUTING: THEORY AND APPLICATIONS, (FICTA 2016), VOL 2, 2017, 516 : 491 - 500
[2] An Approach to Enhance the Performance of Hadoop MapReduce Framework for Big Data
Chandra, Subhash
Motwani, Deepak
2016 INTERNATIONAL CONFERENCE ON MICRO-ELECTRONICS AND TELECOMMUNICATION ENGINEERING (ICMETE), 2016, : 178 - 182
[3] A Hadoop MapReduce Performance Prediction Method
Song, Ge
Meng, Zide
Huet, Fabrice
Magoules, Frederic
Yu, Lei
Lin, Xuelian
2013 IEEE 15TH INTERNATIONAL CONFERENCE ON HIGH PERFORMANCE COMPUTING AND COMMUNICATIONS & 2013 IEEE INTERNATIONAL CONFERENCE ON EMBEDDED AND UBIQUITOUS COMPUTING (HPCC_EUC), 2013, : 820 - 825
[4] Performance Analysis of the Effect of a Combiner on a MapReduce Job
Mhlanga, Imran Artwel J.
Ahmad, Nazrul M.
Azman, Afizan
Razak, Siti Fatimah Abdul
2018 IEEE STUDENT CONFERENCE ON RESEARCH AND DEVELOPMENT (SCORED), 2018,
[5] Combiner to Reduce the Time of Processing in Trend Analysis using Hadoop's MapReduce Framework
Pinto, Vivek Francis
2017 2ND INTERNATIONAL CONFERENCE ON COMPUTATIONAL SYSTEMS AND INFORMATION TECHNOLOGY FOR SUSTAINABLE SOLUTION (CSITSS-2017), 2017, : 166 - 169
[6] Various approches to improve MapReduce performance in Hadoop
Manjaly, Jisha S.
Subbulakshmi, T.
PROCEEDINGS OF THE 2018 3RD INTERNATIONAL CONFERENCE ON INVENTIVE COMPUTATION TECHNOLOGIES (ICICT 2018), 2018, : 778 - 782
[7] Performance Modelling and Analysis of MapReduce/Hadoop Workloads
Yu, Xiaolong
Li, Wei
2015 IEEE 21ST INTERNATIONAL WORKSHOP ON LOCAL & METROPOLITAN AREA NETWORKS (LANMAN), 2015,
[8] Performance analysis of MapReduce Programs on Hadoop cluster
Maurya, Mahesh
Mahajan, Sunita
PROCEEDINGS OF THE 2012 WORLD CONGRESS ON INFORMATION AND COMMUNICATION TECHNOLOGIES, 2012, : 505 - 510
[9] Performance Analysis of Coupling Scheduler for MapReduce/Hadoop
Tan, Jian
Meng, Xiaoqiao
Zhang, Li
2012 PROCEEDINGS IEEE INFOCOM, 2012, : 2586 - 2590
[10] Coupling GPU and MPTCP to Improve Hadoop/MapReduce Performance
Wang, Chia-Hui
Yang, Chen-Kuei
Liao, Wei-Chih
Chang, Ray-I
Wei, Tsao-Ta
2016 2ND INTERNATIONAL CONFERENCE ON INTELLIGENT GREEN BUILDING AND SMART GRID (IGBSG), 2016, : 109 - 114

← 1 2 3 4 5 →