The Gfarm File System on Compute Clouds

Due to its ability to provide requested resources on demand, the use of cloud computing for data-intensive computing is expected to skyrocket in the coming years. The present IaaS cloud infrastructure is designed such that the compute cloud and the storage cloud are separate components. However, the design has one major problem, namely a trade-off has to be made between the network cost and storage performance. To cope with these problems, we are proposing high performance file system service in compute cloud that transparently utilizes file access locality on dynamic configuration of cloud computing infrastructure. We evaluated the system using micro benchmarks, MapReduce applications, and MPI-IO applications. Our system shows the scalable file I/O performance.

I. Introduction

Cloud computing is a promising environment for large scale computing due to its ability to provide necessary CPU and storage resources on demand without any knowledge of the internal infrastructure. Therefore it seems not only be useful for general purpose applications such as web servers, database servers et cetera, but also for scientific applications that require large scale distributed processing such as astronomy and particle physics. That kind of data intensive computing needs a high performance file system that can share data between VMs.