System execution path profiling using hardware performance counters

The task critical execution path, obtained from a kernel trace, reports the time spent waiting for each task involved in a heterogeneous and distributed application. However, additional profiling is needed to understand and identify the problematic code associated with long-lasting path edges. Hardware counter sampling provides insight on software performance at the microarchitecture level, for instance extracting the call stack every 100K execution cycles to understand where the execution time is spent. Similarly, extracting the call stack at the end of a long waiting system call is often useful. This technique is readily available for either statically or JIT compiled code. However, interpreted code is indirectly executed on the processor and the link between the statements and the executed assembly is missing. We propose an architecture to efficiently record call stacks along the execution path, including interpreted programs, in a low intrusive way that maintains the abstraction boundary between the kernel, the interpreter, and the user code. The method consists in sending a signal from within the performance counter interrupt handler. The user-space code receiving the signal can inspect and record the state of the program. We implemented a profiler for the CPython interpreter using this technique. We studied the benefit, the accuracy, and the cost of the proposed technique compared to an all-kernel monitoring solution.