Battle of the Defaults: Extracting Performance Characteristics of HDF5 under Production Load

Popular parallel I/O libraries, such as HDF5, provide tuning parameters to obtain superior performance. However, the selection of effective parameters on production systems is complex due to the interdependence of I/O software and file system layers. Hence, application developers typically use the default parameters and often experience poor I/O performance. This work conducts a benchmarking-based analysis on the HDF5 behaviors with a wide variety of I/O patterns to extract performance characteristics under the production workload. To make the analysis well controlled, we exercise I/O benchmarks on POSIX-IO, MPI-IO, and HDF5 using the same I/O patterns and in the same jobs. To address high performance variability in production environments, we repeat the benchmarks across I/O patterns, storage devices, and time intervals. Based on the results, we identified consistent HDF5 behaviors that appropriate configurations and operations on dataset layout and file-metadata placement can improve performance significantly. We apply our findings and evaluate the tuned I/O library on two supercomputers: Summit and Cori. The results show that our tuned parameters can achieve more than 10× I/O performance speedup than that with default parameters on both systems, suggesting the effectiveness, stability, and generality of our solution.

I. Introduction

In high-performance computing (HPC), many domain scientists manage their data using I/O libraries, such as HDF5 [1], ADIOS [2], PnetCDF [3], etc. These libraries sit between applications and the underlying storage systems, support a variety of data structures, and maneuver toward high through-put across multiple I/O layers. For instance, HDF5 provides parallel I/O services via MPI-IO interfaces, stores files on large-scale parallel file systems, such as Lustre [4] and GPFS [5], and allows end users to select specific configurations for HDF5 internal operations (e.g., HDF5 metadata cache size and management), MPI-IO operations (e.g., through MPI_Info object), and file systems (e.g., data layout properties). We discuss these tunable parameters in details in Section II.