Online Optimization for Machine Learning-based Scheduling in Distributed Computing

In this work, we propose and evaluate an online scheduler prototype based on machine learning algorithms. Online job-flow scheduler should make scheduling and resource allocation decisions for individual jobs without any prior knowledge of the subsequent job queue (i.e., online). We simulate and generalize this task to a more formal 0–1 Knapsack problem with unknown utility functions of the knapsack items. In this way we evaluate the implemented machine learning-based solution to classical combinatorial optimization algorithms. A hybrid machine learning and dynamic programming - based approach is proposed to consider and strictly satisfy the knapsack constraint on the total weight. As a main result the proposed hybrid solution showed efficiency comparable to the greedy knapsack approximation.