State Definition in Deep Reinforcement Learning: A Smoothness Index Approach

In the pursuit of developing autonomous agents capable of learning optimal behaviors, reinforcement learning has emerged as a foundational frame-work. This paper introduces a novel approach for state definition in deep reinforcement learning systems, with a focus on solving pixel-based control scenarios. The proposed frame-work harnesses the capabilities of Convolutional Variational Auto-Encoders to extract compact and meaningful latent representations from high-dimensional pixel observations. These representations serve as the foundation for constructing an efficient state space for reinforcement learning agents. To further enhance the state construction process, the concept of the Smoothness Index is introduced. Smoothness index acts as a criterion for feature selection within the latent space, facilitating the identification of the most relevant and informative states. Demonstrating the effectiveness of iterative smoothness index-based forward selection, this process notably compresses the observation space while retaining essential information. This compression enables the creation of more efficient reinforcement learning agents. The procedure involves collecting and preprocessing observation data given by the OpenAI Gym car racing environment, encoding observations using a convolutional variational auto-encoder, and iteratively selecting features based on smoothness index. Subsequently, the Proximal Policy Optimization algorithm is employed as a policy-based controller to evaluate the effectiveness of the proposed frame-work. Results indicate that this approach yields a remarkably compact state space that either maintains or enhances reinforcement learning agent performance. This innovative frame-work paves the way for the development of more efficient and adaptive reinforcement learning agents in real-world image-based applications.

I. Introduction

In the field of control and robotics, one of the primary objectives of artificial intelligence is to create fully autonomous agents capable of interacting with their environment, learning optimal behaviors, and continually improving through trial and error over time. This challenge spans a wide spectrum, ranging from robots with sensory inputs doing a task to software-based agents engaging in natural language and multimedia interactions [1].