Applying Machine Learning to Understand Write Performance of Large-scale Parallel Filesystems

In high-performance computing (HPC), I/O performance prediction offers the potential to improve the efficiency of scientific computing. In particular, accurate prediction can make runtime estimates more precise, guide users toward optimal checkpoint strategies, and better inform facility provisioning and scheduling policies. HPC I/O performance is notoriously difficult to predict and model, however, in large part because of inherent variability and a lack of transparency in the behaviors of constituent storage system components. In this work we seek to advance the state of the art in HPC I/O performance prediction by (1) modeling the mean performance to address high variability, (2) deriving model features from write patterns, system architecture and system configurations, and (3) employing Lasso regression model to improve model accuracy. We demonstrate the efficacy of our approach by applying it to a crucial subset of common HPC I/O motifs, namely, file-per-process checkpoint write workloads. We conduct experiments on two distinct production HPC platforms - Titan at the Oak Ridge Leadership Computing Facility and Cetus at the Argonne Leadership Computing Facility - to train and evaluate our models. We find that we can attain ≤ 30% relative error for 92.79% and 99.64% of the samples in our test set on these platforms, respectively.

I. Introduction

HPC productivity could be greatly enhanced by more accurate I/O performance prediction. At supercomputing facilities, scientists are allocated limited compute cycles and charged based on their application execution times. They are therefore motivated to limit or reduce the time consumption on I/O. For the facilities, more predictable I/O performance will enable more precise compute-time allocations and further enable additional jobs on the same platform, increasing the system utilization. For example, for many scientists the ideal time spent on writing state snapshots is ~10% of the total application execution times. In practice, however, the cost may span a wide range (e.g., 7%-20% for XGC code [39]) because of lack of guidance on I/O configuration. With accurate write performance prediction, scientists can manage the cost by configuring the write frequencies appropriately (§II-A).