Integrating Mechanism and Data: Reinforcement Learning Based on Multi-Fidelity Model for Data Center Cooling Control

When dealing with practical Reinforcement Learning (RL) problems in the data center (DC) cooling system, a common approach is to conduct policy training in high-fidelity simulator environments (e.g, a computational fluid dynamics (CFD) simulator), and subsequently deploy the trained policy to the real-world environment. However, utilizing high-fidelity simulators for simulation demands enormous computational resources and time, and the sample inefficiency of existing RL algorithms exacerbates this challenge. To address this problem, we propose a sample-efficient RL framework designed for green DC cooling control. To achieve this, we integrate both physical mechanisms and data-driven approaches to create a bi-fidelity RL environment. Specifically, the RL agent learns fundamental system knowledge from the low-fidelity model, while the high-fidelity model is employed to provide further details for policy training. The agent's understanding level of the state-action space is measured based on physical mechanisms, and this measurement serves as the criterion for switching between the multi-fidelity models. Experimental results demonstrate that the proposed framework significantly reduces the training time of RL policy without sacrificing the performance, compared to training the policy exclusively in the high-fidelity CFD simulator.