Enhanced Resource Management Enabling Standard Parameter Sweep Jobs for Scientific Applications

Parameter sweeps are used by researchers with scientific domain-specific tools or workflows to submit a large collection of computational jobs whereby each single job of it only varies in certain parts. They require a more fine-grained distribution of jobs across resources, which also raise a significant challenge for efficient resource management in middleware environments that have been not specifically designed to perform parameter sweeps. This paper offers insights into parameter sweep solutions that support multi-disciplinary science environments via abstraction from resource management complexities using middleware. The solutions are based on use case requirements, enable efficient submission, enhanced usability, and standard compliance. We also apply a use case taken from the life science domain to demonstrate usefulness and efficiency of the solutions.

I. Introduction

For scientific experimentation and analysis the efficient job and data management is considered to be the most significant functionality. In most of the cases the applications are being accessed or executed through a layer of middleware abstraction. This abstraction can have a potential to interface with multiple computing and data platforms. The computing platforms here could be either high performance computing (HPC) or high throughput computing (HTC) resources. We have witnessed in different scientific domains that the use of middleware abstraction over the mentioned resource management systems is becoming useful.