Design of flexible Compression Transmission Framework for Distributed Storage System Based on Erasure Coding

Erasure Coding is a redundancy technology commonly used in distributed storage systems (DSS) to ensure data reliability. However, the large number of repair operations required by Erasure Coding results in a large consumption of network resources, which constitutes a performance bottleneck in Erasure Coding storage systems. In this paper, we design and implement a new distributed storage framework FCT-EC, which the framework decouples Erasure Coding operations from the original distributed storage system, monitors cluster network congestion, and establishes a transmission revenue model to help evaluate and select the best compressed transmission method to alleviate network congestion in Erasure Coding storage systems. Extensive experiments in real environments show that FCT-EC maintains normal write and read performance. When network resources are limited, FCT-EC accelerates repair performance by 12.99-52.93%.