A Compression Framework for Multidimensional Scientific Datasets

Scientific simulations and instruments can generate tremendous amount of data in short time periods. Since the generated data is used for inferring new knowledge, it is important to efficiently store and provide it to the scientific endeavors. Although parallel and distributed systems can help to ease the management of such data, the transmission and storage are still challenging problems. Compression is a popular approach for reducing data transfer overheads and storage requirements. However, effectively supporting compression for scientific simulation data and integrating compression algorithms with simulation applications remain a challenge. In this work, we focus on management of multidimensional scientific datasets using domain specific compression algorithms. We propose a compression framework and methodology in order to maximize the bandwidth and storage utilization. We port our framework into PnetCDF and present our preliminary experimental results.