A Simple Approach to Optimize S3 Object Gateways for Massive Numbers of Small File Writes

This paper shows an approach to optimizing S3 Object Gateways for massive number of small file writes by combining database and file system technology. The basic concept is to route small data to a Wide Column Store database and big files to a distributed file system. This approach helps to overcome the limitation of distributed file systems which are commonly used as the storage backend for S3 Object Gateways. We have shown the technical feasibility of this concept based on an experiment for write operations by modifying an open-source S3 Object Gateway with an integrated routing mechanism; files equal or greater than 500 KiB are written to the HPE Data Fabric Distributed File System and files smaller than 500 KiB as well as object metadata are written to the HPE Data Fabric Database using the HBase-API. Our experiment using the Wassabi S3 Benchmark tool shows that performance can be significantly improved compared to cloud-based S3 Object Storage systems and we measured approximately 30,000 PUTs per second for files of 10 KiB in our test bed.