The Dynamic Replication Mechanism of HDFS Hot File based on Cloud Storage

As an open source cloud storage scheme, HDFS is used by more and more large enterprises and researchers, and is actually applied to many cloud computing systems to deal with huge amounts of data. HDFS has many advantages, but there are some problems such as NameNode single point of failure, small file problem, hot issues, etc. For HDFS hot issues, this paper proposes a dynamic Replication mechanism of HDFS hot file based on cloud storage(HDFS-DRM). The mechanism includes a Replication of the dynamic adjustment mechanism and adding, deleting duplicate node selection mechanism in two parts, by increasing the NameNode, BlockMap parameters, it records the number of reading requests of each file in a certain period of time to decide whether to increase or decrease the number of copies. The mechanism presents a replica placement method based on stage historical information and node load and selects the appropriate node to add or delete copies of documents to improve the utilization efficiency of the data node storage space effectively. Experimental results show that, HDFS - DRM in hot files case, compared to native HDFS file system access latency is significantly reduced, HDFS-DRM can solve the hot issues successfully.