Mining Scientific Data using the Internet as the Computer

This paper describes approaches and methodologies facilitating the analysis of large amounts of distributed scientific data. The existence of full-featured analysis tools, such as the Algorithm Development and Mining (ADaM) toolkit and online data repositories now provide easy access and analysis capabilities to large amounts of data. However, there are obstacles to getting the analysis tools and the data together in a workable environment. Does one bring the data to the tools or deploy the tools close to the data? The large size of many current Earth science datasets incurs significant overhead in network transfer for analysis workflows, even with the current advanced networking capabilities. We are developing two solutions for this problem that address different analysis scenarios. The first is a Data Center Deployment of the analysis services for large data selections, orchestrated by a remotely defined analysis workflow. The second is a Data Mining Center approach of providing a cohesive analysis solution for smaller subsets of data. The two approaches can be complementary and thus provide flexibility for researchers to exploit the best solution for their data requirements.

I. INTRODUCTION

The NASA-funded project titled “Deployable Suite of Data Mining Web Services for Online Science Data Repositories” is developing technology to allow users to locally define analysis workflows that can be executed on data residing in online repositories. The objective of this technology is to implement web-based data mining and image processing services, along with general purpose data preparation utilities, that can be deployed with ease by different distributed online data providers and used immediately in data processing and analysis activities. These services form building blocks for distributed, flexible processing in a Service-Oriented Architecture (SOA) built around web services, an emerging paradigm of software design and deployment. Such an architecture allows different groups that focus on science data processing, basic research and the applications arena to chain the mining services together with other services and community-based toolkits in many different ways.