Many-Constraint and Many-Objective optimization with Bias Index for Intercloud Multi-Workflow Resource Provisioning

Optimal deployment of multi-component big data applications is difficult in geo-distributed intercloud environments. This is because of the numerous infrastructure components and options available and the variety of constraints that must be satisfied, such as application, cloud infrastructure, data processing, and privacy-related constraints. The task becomes even more complicated when multiple scientific workflows must be executed with a severely limited resource-acquisition budget. This paper proposes a many-objective constrained optimization framework that addresses these issues. The proposed framework first conducts constraint satisfaction via equivalent transformation, then many-objective optimization using nondominated sorting, reference points, and elitism. In the case of multiple workflows, both optimization for each workflow and optimization for the ensemble of workflows are considered. Furthermore, a proposed bias index is presented that indicates on an objective-by-objective basis the effect of the configuration generated for each ensemble of workflows on the optimal configuration of each constituent workflow. It also provides a means of ascertaining on a granular level the relative fairness of each objective in each composite resource configuration, and can be used as a tool for prioritizing certain aspects of a workflow when deciding on the optimal configuration to utilize. We demonstrate the efficacy of the proposed framework through two genome analysis workflows, for which site availability and resource reliability need to be maximized, deployment cost and makespan need to be minimized, and several constraints must be satisfied.