An Implementation for Secure Data Deduplication on End-to-End Encrypted Documents

In the realm of data storage and management, secure data deduplication represents a cornerstone technology for optimizing storage space and reducing redundancy. Traditional client-side deduplication approaches, while efficient regarding storage and network traffic, expose vulnerabilities that allow malicious users to infer the existence of specific files through traffic analysis. Even using a Proof of ownership scheme does not guarantee protection from all attack scenarios, specific to data deduplication. This paper introduces a novel secure data deduplication framework employing a deduplication proxy that operates on-premise, effectively mitigating the risk of such inference attacks. By leveraging convergent encryption, and Merkle tree chal-lenges for proof of ownership, our solution ensures that data deduplication does not compromise data privacy or security. The deduplication proxy acts as an intermediary, performing dedu-plication processes on-premise. This approach not only preserves the efficiency benefits of deduplication but also enhances security by preventing external visibility into data traffic patterns. Our implementation, publicly available on Github, demon-strates the efficacy of the method for enforcing end-to-end encryption while maintaining data deduplication's storage-saving advantages. The proposed framework is suitable for organizations aiming to safeguard their data while optimizing storage resources.