The Grid2003 production grid: principles and practice

The Grid2003 Project has deployed a multivirtual organization, application-driven grid laboratory ("Grid3") that has sustained for several months the production-level services required by physics experiments of the Large Hadron Collider at CERN (ATLAS and CMS), the Sloan Digital Sky Survey project, the gravitational wave search experiment LIGO, the BTeV experiment at Fermilab, as well as applications in molecular structure analysis and genome analysis, and computer science research projects in such areas as job and data scheduling. The deployed infrastructure has been operating since November 2003 with 27 sites, a peak of 2800 processors, work loads from 10 different applications exceeding 1300 simultaneous jobs, and data transfers among sites of greater than 2 TB/day. We describe the principles that have guided the development of this unique infrastructure and the practical experiences that have resulted from its creation and use. We discuss application requirements for grid services deployment and configuration, monitoring infrastructure, application performance, metrics, and operational experiences. We also summarize lessons learned.

1 Introduction

The Grid2003 Project [1] has deployed for the first time a persistent, shared” multi-virtual organization (VO) [2], multi-application grid laboratory capable of providing production level services for large‥ scale computation-and data-intensive science applications. The project was organized by representatives of the U.S. “Trillium” projects (the GriPhyN virtual data research project [3], Particle Physics Data Grid, PPDG [4], International Virtual Data Grid Laboratory, iVDGL [5]) and the U.S. ATLAS [6] and U.S. eMS [7] Software and Computing Projects of the Large Hadron Collider (LHe) [8] program at CERN [9]. The goal of Grid2003 was to build an application grid laboratory (“Grid3”) that would provide: •

A platform for experimental computer science research by GriPhyN and other grid researchers;.

•

The infrastructure and services needed to demonstrate LHC production and analysis applications running at scale in a common grid environment;.

•

The ability to support multiple application groups, including the Sloan Digital Sky Survey (SDSS) [10] and the Laser Interferometer Gravitational Wave Observatory (LIGO) [11], [12], core participants in GriPhyN and iVDGL.