IDaPS - Improved data-locality aware data placement strategy based on Markov clustering to enhance MapReduce performance on Hadoop

The execution of Map-Reduce applications on the Hadoop cluster poses significant challenges due non-consideration of data locality, i.e., assigning tasks to compute nodes where input data sets are Due to such non-consideration, high data transfer overheads are caused. Further, it increases latency, may arise if input data needs to be transferred across the network, thereby significantly increasing execution time. To address this issue, an Improved DAta Placement Strategy IDaPS based on the intra-dependency among the data is proposed. IDaPS re-organizes the default data layouts in HDFS to ensure higher of parallelism. The efficiency of IDaPS is demonstrated in Hadoop clusters (10 and 15 nodes) by executing Hadoop Benchmark performance tests viz. WordCount, Grep on Project-Gutenberg book dataset (50 Least Square Linear Regression (LSLR) on weather dataset (10.67 GB). The results were compared with of-the-art algorithms viz. Hadoop Default Data Placement (HDDP), Load-Balancer and literary work The results demonstrate that IDaPS significantly reduces execution time by 28.2% and 38.4% in 10-node 15-node clusters while executing WordCount, and 35% and 38.1% in 10-node and 15-node clusters for Similarly, for LSLR, it reduces execution time by 32.7%.