HDFS Improvement Using Shortest Path Algorithms

In the previous version of the article "data processing on distributed systems: storage challenge" we presented a new approach for the storage, management and exploitation of distributed data in the form of small files, such as the case of message exchanges real-time localization in port activity. During this approach, we managed to optimize more than 30% of information management using the classic HADOOP/YARN/HDFS architecture [11]. Considering that in aHADOOPecosystem with several data processing nodes, access to the right node, containing the desired data, in the optimal time presents a major challenge and very important research avenues for researchers and scientists [15]. In this paper, we will see together that the marriage between mathematical algorithms and computer magic can give us very encouraging and very important results. Indeed, one of the principle that manifests itself is the theory of graphs, especially the calculation of the shortest path to optimally reach the data on a few nodes in an architecture of a few hundred nodes or even thousands [16]. After several research and comparison, Dijkstra's algorithm is the chosen algorithm for calculating the shortest path in a HADOOP/HDFS system.