Write Bandwidth Optimization of Online Erasure Code Based Cluster File System

As the data volume is growing from big to huge in many science labs and data centers, more and more data owners are willing to choose Erasure Code based storage to reduce the storage cost. However, online Erasure Code based cluster file systems still have not been applied widely because of write bottlenecks in data encoding and data placement. We proposed two optimizations to address them respectively. We propose a Partition Encoding policy to accelerate the encoding arithmetic through SIMD extensions and to overlap data encoding with data committing. We devise Adaptive Placement policy to provide incremental expansion and high availability, as well as good scalability. The experimental results in our prototype ECFS show that the aggregate write bandwidth can be improved by 42%, while keeping the storage in a more balanced state.