Blaze: A High-Performance, Scalable, and Efficient Data Transfer Framework with Configurable and Extensible Features : Principles, Implementation, and Evaluation of a Transatlantic Inter-Cloud Data Transfer Case Study

Blaze is a high-speed data transfer framework that enables efficient and scalable data movement between distributed storage systems. In this paper, we describe the design, implementation, and evaluation of Blaze in the context of a case study involving the transfer of 5.6 petabytes of data from OVH cloud storage to an Amazon S3 bucket. We discuss the technical challenges and design choices that led to creating a single-agent architecture, providing flexibility in agent placement while minimizing operational costs. We also demonstrate the orchestration capabilities of Blaze using Apache Airflow to manage hierarchical workflows for data transfer between storage systems. Our evaluation shows that Blaze achieved the expected 20 Gbps throughput during data transfer and provided significant cost savings compared to other architectures. The results demonstrate that Blaze is a practical solution for high-speed data transfer in large-scale distributed storage environments.

I. Introduction

Efficiently transferring large volumes of data across dis-tributed storage environments has become increasingly important as data sizes and cloud adoption continue to grow. Today, we commonly encounter a mix of on-premise and multi-cloud environments, making the need for high-performance, scalable, and efficient data transfer frameworks even more critical. Transferring data across different cloud platforms with proprietary protocols can be particularly daunting, especially involving data transfers across continents.