Erasure Coding and Data Deduplication: A Comprehensive Survey

Managing and organizing data is an ever-growing challenge in the digital age, and this literature survey takes us on a journey through the key strategies and approaches that are shaping the landscape of data storage and management. The survey begins with a look at fixed-length chunking, a seemingly straightforward method for dividing data. While it offers speed and simplicity, it also grapples with the "byte shifting problem". Content-defined chunking algorithms step in as the solution, adapting to the data’s content, and overcoming this limitation. Variable-length chunking algorithms, like the Rabin-based Algorithm and MII, provide flexibility to address different data scenarios, from minor changes to major ones. In this paper, we discussed various content defined chunking algorithms and their performance based on chunking properties like chunking speed, processing time, and throughput. Content-Defined Chunking (CDC) techniques, such as Fast CDC, MII, and SS-CDC, help streamline data management, minimizing redundancy efficiently.Delta encoding, general compression, and data deduplication play essential roles in reducing data size. Gdelta, a novel approach, strikes a balance between these techniques, enhancing data reduction.