Demystifying Complex Treatment Recommendations: A Hierarchical Cooperative Multi-Agent RL Approach

Reinforcement Learning (RL) techniques are widely adopted in healthcare research to learn effective policies for complex sequential decision-making tasks, such as treatment recommendation to achieve better clinical outcomes. However, existing RL based solutions suffer from two major limitations. (1) Burdens a single agent to make complex treatment recommendations: existing RL algorithms do not explicitly support the common hierarchical decision-making processes used in practise. That is, they consider treatment recommendation as a singular decision and the treatment space as a flat search space, and expect a single RL agent to select the best treatment outright. (2) Sub optimal patient state representation techniques: accurate representation of patient states and their dynamics is essential to make effective clinical decisions. However, there are three main limitations in current state representation techniques in the clinical environment: (i) lack of clear structure and high variability of input features; (ii) lack of a temporal context, which makes it difficult to fully capture patient conditions and their dynamics at a given time; (iii) and the use of imputation techniques, which undermines personalization, introduce biases and errors that can cause harm to patients. To address the complexities of treatment recommendation and generate superior treatment policies, we propose a consolidated RL solution that incorporates a hierarchical multi-agent architecture and advanced patient state representation technique. In our proposed architecture, the complex task is decomposed into a hierarchy of subtasks, each handled by a specialized agent that coordinates skills learned by lower level agents to determine a high-level treatment policy. The patient state representation technique is designed to effectively encode the changing states of patients throughout their hospital stay, addressing the three primary limitations of current approaches. Our experiments using the sepsis treatment task shows that the proposed solution consistently outperforms state-of-the-art baselines and improves patient survival.