Novelty Search for Neuroevolutionary Reinforcement Learning of Deceptive Systems: An Application to Control of Colloidal Self-assembly

Colloidal self-assembly systems are generally difficult to control due to their highly nonlinear and stochastic dynamics and sparse rewards. These systems are also inherently deceptive, as successful control policies must be able to smooth out unavoidable defects and therefore temporarily move farther away from their goal in order to eventually realize the desired goal. This paper investigates the viability of evolutionary reinforcement learning (RL) based on novelty search, wherein behavioral novelty alone is used to learn control policies that can systematically mitigate deceptive dynamics. As such, for stochastic nonlinear systems that are prone to a deceptive behavior, novelty search is a promising alternative to the widely used objective search RL, where merely progress towards a pre-defined goal is used to learn and update control policies. In this work, we pair novelty search RL with a complexifying algorithm that simultaneously learns the neural network architecture and parameters of a control policy. The complexifying algorithm principles the novelty search by ensuring that simple behaviors must be discovered before more complex ones. We evaluate the performance of evolutionary RL based on objective search and novelty search on a benchmark in-silico colloidal self-assembly problem.