An AI-aware Orchestration Framework for Cloud-based LLM Workloads

Large Language Models (LLMs), such as ChatGPT and Bard, have revolutionized the entire industry with their extraordinary power, and gained traction in the cloud community. Deploying LLM workloads in cloud has become increasingly common nowadays, as cloud easily fulfills their requirement on extensive computing resources. There are already tools (such as Kubernetes) that help simplify the management of Artificial Intelligence (AI) workloads, yet LLMs often require the balancing of multi-dimensional demands, which can be hard to handle by basic tools. This paper introduces an AI-aware orchestration framework that optimizes LLM performance from aspects including scheduling, scaling, and advanced data management. It can improve the cost-efficiency of the LLM workloads while ensuring its performance, while ensuring the absence of common security risks, supported by Intel hardware.