Error-controlled Progressive Retrieval of Scientific Data under Derivable Quantities of Interest

The unprecedented amount of scientific data has introduced heavy pressure on the current data storage and transmission systems. Progressive compression has been proposed to mitigate this problem, which offers data access with on-demand precision. However, existing approaches only consider precision control on primary data, leaving uncertainties on the quantities of interest (QoIs) derived from it. In this work, we present a progressive data retrieval framework with guaranteed error control on derivable QoIs. Our contributions are three-fold. (1) We carefully derive the theories to strictly control QoI errors during progressive retrieval. Our theory is generic and can be applied to any QoIs that can be composited by the basis of derivable QoIs proved in the paper. (2) We design and develop a generic progressive retrieval framework based on the proposed theories, and optimize it by exploring feasible progressive representations. (3) We evaluate our framework using five real-world datasets with a diverse set of QoIs. Experiments demonstrate that our framework can faithfully respect any user-specified QoI error bounds in the evaluated applications. This leads to over $2.02 \times$ performance gain in data transfer tasks compared to transferring the primary data while guaranteeing a QoI error that is less than 1E-5.

I. Introduction

The arrival of the first generation of exascale machines and the continuous upgrading of experimental and observational facilities have presented a huge strain on storage, I/O, and networking due to unprecedented data volume and velocities. Because of the limited spacing at high-end parallel file systems (PFS), most of these data must be moved to lower-tier storages, such as tapes, right after generation. Future analyses, which require retrieving data from a central repository and moving across wide area networks, must consider the cost of data retrieval and movement. Recently, lossy compression methods [1]–[5] have been developed to tackle the I/O and storage bottleneck as they demonstrate greater compressibility than lossless compressors on floating-point scientific data. Since most simulation and experimental devices inherently involve uncertainty and variability, data can be reduced, provided the loss of accuracy is under prescribed bounds.