Differential Performance Analysis Workflow for Algorithmic Changes

Most performance analysis tools used in HPC focus on the analysis of a single configuration of an application. In this work, we instead present a novel performance analysis workflow, supporting the comparison of varied code versions and running conditions. There exist different code versions for many applications because they comprise parts that can be implemented in various ways or already exist in third-party libraries, like linear solvers. Additionally, varied running conditions like scaling of execution units or exchanging the input data can influence the performance behavior. Performance comparison of different application configurations helps determine the best configuration and understand differences in behavior. Such measurements are often not supported directly and are cumbersome to handle manually with current performance measurement and analysis tools. This work presents a workflow based on the JUelich Benchmarking Environment (JUBE) that automatically handles the multitude of measurements and data collation after an initial manual configuration. Furthermore, we introduce diagrams suited for a clear and precise presentation of the collected performance data. The proposed workflow is showcased using two applications CalculiX and Jukkr. Our application studies highlight that our workflow allows a detailed performance analysis while still being easy to use. We, therefore, encourage integrating our approach of multi-configuration diagrams into broadly used HPC visual performance exploration tools.