Blaze: A High-Performance, Scalable, and Efficient Data Transfer Framework with Configurable and Extensible Features

Blaze is a high-speed data transfer framework that enables efficient and scalable data movement between distributed storage systems. In this paper, we describe the design, implementation, and evaluation of Blaze in the context of a case study involving the transfer of 5.6 petabytes of data from OVH cloud storage to an Amazon S3 bucket. We discuss the technical challenges and design choices that led to creating a single-agent architecture, providing flexibility in agent placement while minimizing operational costs. We also demonstrate the orchestration capabilities of Blaze using Apache Airflow to manage hierarchical workflows for data transfer between storage systems. Our evaluation shows that Blaze achieved the expected 20 Gbps throughput during data transfer and provided significant cost savings compared to other architectures. The results demonstrate that Blaze is a practical solution for high-speed data transfer in large-scale distributed storage environments.