Proactive Resource Management to Optimize Distributed Workflow Executions

Scientific workflows have received increasing interest and are used in many scientific fields to gather, analyze, and process significant amounts of data. However, their tasks are usually treated as black boxes, and their behavior remains unconsidered for resource allocations, which can lead to subpar resource allocations with typical scheduling. Although not done yet, it should be possible to observe such tasks, learn their behavior, and use this knowledge to improve future executions. As workflows and their tasks are often executed multiple times on a massive scale, even a slight improvement per execution may save hours of execution time and significant amounts of energy.To achieve this goal, we develop an innovative approach to model task executions and predict resource usage. The prediction is embedded in a feedback loop to repeatedly improve the models and to closely track workflow executions to make predictions and resource allocations accurate.