ROIL: Rule Optimization via Large Language Model for Imitation Learning

Recent improvements in pretrained Large Language Models (LLMs) have demonstrated increasing capabilities in various natural language processing tasks, such as instruction following. However, effectively utilizing LLMs in interactive environments that require advanced reasoning, planning, and decision-making skills remains a challenge. In this study, we integrate Learning Classifier Systems (LCS) with LLMs, thereby extending their capabilities to interact in natural language domains. To accommodate the integration, especially the rule represented in natural language, we propose a novel Rule Optimization method using LLMs for Imitation Learning (ROIL) in text-based interactive environments. ROIL addresses rule learning by lever-aging LLMs to optimize rules based on human demonstrations, thus eliminating the need for trial-and-error learning processes and ensuring both interpretability and safety. It also tackles the challenge of learning transferable skills in these environments. We evaluate the efficacy of ROIL in the WebShop environment, a text-based e-commerce website navigation problem. The method achieved a significant performance and efficiency improvement over a strong baseline, while demonstrating performance close to a gradient-based imitation learning approach. Additionally, this study explores the enhancement of ROIL using meta-heuristic optimization algorithms, providing foundational research for further investigation in this area.