Application-Driven Requirements for Node Resource Management in Next-Generation Systems

Emerging workloads on supercomputing platforms are pushing the limits of traditional high-performance computing software environments. Multi-physics, coupled simulations, big data processing and machine learning frameworks, and multi-component workloads pose serious challenges to system and application developers. At the heart of the problem is the lack of cross-stack coordination to enable flexible resource management among multiple runtime components.In this work, we analyze seven real-world applications that represent emerging workloads and illustrate the scope and magnitude of the problem. We then extract several themes from these applications that highlight next-generation requirements for node resource managers. Finally, using these requirements, we propose a general, cross-stack coordination framework and outline its components and functionality.

I. Introduction

In recent years workload diversity in HPC environments has exploded. Big data processing, in-situ analytics, artificial intelligence and machine learning workloads, as well as multi-component workflows are becoming common-place on supercomputers. Each of these workloads bring specific runtime requirements that are pushing the traditional supercomputing software environments to their limits.