Big Data Challenges in Climate Science: Improving the next-generation cyberinfrastructure

The knowledge we gain from research in climate science depends on the generation, dissemination, and analysis of high-quality data. This work comprises technical practice as well as social practice, both of which are distinguished by their massive scale and global reach. As a result, the amount of data involved in climate research is growing at an unprecedented rate. Some examples of the types of activities that increasingly require an improved cyberinfrastructure for dealing with large amounts of critical scientific data are climate model intercomparison (CMIP) experiments; the integration of observational data and climate reanalysis data with climate model outputs, as seen in the Observations for Model Intercomparison Projects (Obs4MIPs), Analysis for Model Intercomparison Projects (Ana4MIPs), and Collaborative Reanalysis Technical Environment-Intercomparison Project (CREATE-IP) activities; and the collaborative work of the Intergovernmental Panel on Climate Change (IPCC). This article provides an overview of some of climate science's big data problems and the technical solutions being developed to advance data publication, climate analytics as a service, and interoperability within the Earth System Grid Federation (ESGF), which is the primary cyberinfrastructure currently supporting global climate research activities.

Technical Challenges for Big Data

The term big data is used to describe data sets that are too large or complex to be worked with using commonly available tools [1]. Climate science represents a big data domain that is experiencing unprecedented growth [2]. Some of the major big data technical challenges facing climate science are easy to understand. ◗

Large repositories mean that the data sets themselves cannot easily be moved; instead, analytical operations must migrate to where the data reside.

◗

Complex analyses over large repositories require high-performance computing.

◗

Large amounts of information increase the importance of metadata, provenance management, and discovery.

◗

Migrating codes and analytic products within a growing network of storage and computational resources creates a need for fast networks, intermediation, and resource balancing.

◗

Importantly, the ability to respond quickly to customer demands for new and often unanticipated uses for climate data requires greater agility in building and deploying applications [3].