A Hybrid Cloud Framework for Scientific Computing

Cloud services are transforming many computing tasks, but the unique requirements of scientific computing have caused it to lag behind in cloud adoption because of the performance variation of cloud resources. Based on our experience with the Organic Grid, we propose a framework for a hybrid cloud that will intelligently distribute work to appropriate computing resources to mitigate the impact of performance variation. We describe a cloud framework that integrates with specialized hardware and distributes work intelligently among heterogeneous computing resources. Our approach is to organize a set of computing nodes in an overlay network, to allow each node as an individual agent to position itself within the network to maximize its productivity. An application finds the resources and decides which task to run on which cloud nodes. Our simulations demonstrate that our methods can significantly reduce communication burdens of the most overworked nodes, especially on networks with the highest task to node ratios.

1. Introduction

Many scientific fields, such as quantum chemistry, genomics, phylogenetics, astrophysics, geophysics, computational neuroscience, or bioinformatics, require massive computational power and resources that might exceed those available on a single local supercomputer. Historically, there have been two drastically different approaches for harnessing the combined resources of a distributed collection of machines: traditional computational supercomputer grids and large-scale desktop-based master-worker grids. Over the last decade, a commercial alternative has become available with cloud computing. For many smaller scientific applications, using cloud computing resources might be cheaper than maintaining a local supercomputer. However, cloud computing has not yet been successful for high-performance computing applications.