Dynamic Workflow Scheduling in the Edge-Cloud Continuum: Optimizing Runtimes Under Budget Constraints

Scientific workflows are increasingly adopting hy-brid Edge-Cloud infrastructures to benefit from the computational and storage capacity of the Cloud and the cost savings and data locality of the Edge. Workflow scheduling is one of the most challenging problems for the Edge-Cloud continuum. State-of-the-art workflow schedulers often rely on a centralized runtime system and are based on static algorithms that either focus on Cloud or Edge systems (but not both). In this paper, we introduce a novel, open-source, and dynamic scheduler for scientific workflows that targets the Edge-Cloud continuum by design using fully decentralized runtime system instances. This not only reduces data transfer times but also leverages the benefits of the continuum. The proposed scheduler optimizes for runtime while adhering to a given cost limit by dynamically mapping tasks to resources and orchestrating groups of workflow tasks on runtime system instances. Furthermore, the scheduler adapts to real-time updates in task durations, accommodating for variations in resource performance, to efficiently use the cost limit and to reduce the total runtime of the workflow. We compare our approach against a state-of-the-art dynamic scheduler (JIT-C) for four well-known scientific workflows. Experiments demonstrate that by using the smallest cost derived by JIT-C as a cost limit for our scheduler, we achieve an average runtime improvement of 56% and an average cost reduction of 34% compared to JIT-C.