GATES: a grid-based middleware for processing distributed data streams

Increasingly, a number of applications rely on, or can potentially benefit from, analysis and monitoring of data streams. Moreover, many of these applications involve high volume data streams and require distributed processing of data arising from a distributed set of sources. Thus, we believe that a grid environment is well suited for flexible and adaptive analysis of these streams. This paper reports the design and initial evaluation of a middleware for processing distributed data streams. Our system is referred to as GATES (grid-based adaptive execution on streams). This system is designed to use the existing grid standards and tools to the extent possible. It flexibly achieves the best accuracy that is possible while maintaining the real-time constraint on the analysis. We have developed a self-adaptation algorithm for this purpose. Results from a detailed evaluation of this system demonstrate the benefits of distributed processing, and the effectiveness of our self-adaptation algorithm.

1 Introduction

The emergence of grids is providing an unprecedented opportunity to solve problems involving very large datasets. However, the existing work in this area has so far focused on static datasets resident in data-repositories [14]. Increasingly, a number of applications across computer sciences and other science and engineering disciplines rely on, or can potentially benefit from, analysis and monitoring of data streams.