Thread-Aware Cache Simulator for HPC Application Tuning

In this paper, we present a thread-aware and parallel cache simulator to assist a performance tuning workflow of HPC applications. Our simulator mimics the behavior of multilevel cache of Fujitsu A64FX processor and enables to detect conflict misses of caches. Since cache conflict misses are known to be reduced by tuning techniques such as array padding, the detection of conflict misses is very useful for tuning HPC applications. To speed up simulations for multi-threaded code, we investigate a design space of the online cache simulator implemented using a dynamic binary instrumentation framework. To model multiple L1 data caches within a single CPU and map them to threads, we build two designs: 1) monolithic simulator, where all of L1 data cache simulations for a CPU are independently performed using a separated process by message passing communication of observed memory traces, 2) parallel simulator, where an L1 data cache simulation is performed in the same thread running the original application as instrumentation code. We evaluate the accuracy and the runtime of our simulator using multi-threaded benchmark programs on the machine with an A64FX processor. By comparing the number of memory references and L1 data cache misses with Fujitsu profiler, we confirm that our simulator can accurately model the underlying cache behavior. The evaluation results show that the parallel simulator design has scalability according to the number of threads assigned for each application, and its execution time is always shorter than that of the monolithic simulator design. These results indicate that the parallel simulator design is promising for assisting a performance tuning workflow of multi-threaded HPC applications.