Training Microrobots via Reinforcement Learning and a Novel Coding Method

Microswimmers have promising applications in different fields, especially in biomedical branches, like microsurgery, therapeutic delivery, and driving sperm cells. Many researchers have proposed different synthetic microrobots with various mechanisms and effective propulsion strategies. Based on the Scallop Theorem, only special sequences of movements cause an effective swimming, so finding these sequences are important. Researchers have been striving to improve microrobots’ propulsion strategies and find new ways to reach optimal propulsion tactics. One of the excellent approaches is using Artificial Intelligence branches, like Reinforcement Learning, to train microrobots. In this study, we introduce a novel coding method called "Binary Coding," which can be utilized in applying Reinforcement Learning algorithms to different microrobots. In this work, we train linear microrobots consisting of spheres and extensible connecting rods between them via Binary Coding and two Reinforcement Learning algorithms, Q-Learning and Expected Value SARSA. For this purpose, we consider several episodes, each of which has sufficient learning steps. By selecting appropriate values for learning parameters, the smart microrobots learn the optimal propulsion cycle (sequence of movements) by themselves and generate a large net displacement in the final episode.

I. Introduction

In the motion of microscale swimmers, a group of microrobots on which we study in this research, the effects of inertia are negligible compared to the viscous effects [1]. In such a regime, based on Purcell’s theorem of microscopic swimming (Scallop Theorem), swimming strategies can be effective and successful only if they involve a cyclic and non-time-reversible motion [1]. Therefore, the usual swimming mechanisms we know for humans or other macroscopic swimmers are ineffective for microscopic swimmers. Many researchers have proposed different synthetic microrobots, with their effective propulsion strategies, which can move in one [2], two [3], or three dimensions [4].