Soil-Adaptive Excavation Using Reinforcement Learning

In this letter, we present an excavation controller for a full-sized hydraulic excavator that can adapt online to different soil characteristics. Soil properties are hard to predict and can vary even within one scoop, which requires a controller that can adapt online to the encountered soil conditions. The objective is to fill the bucket with excavation material while respecting machine limitations to prevent stalling or lifting of the machine. To this end, we train a control policy in simulation using Reinforcement Learning (RL). The soil interactions are modeled based on the Fundamental Equation of Earth-Moving (FEE) with heavily randomized soil parameters to expose the agent to a wide range of different conditions. The agent learns to output joint velocity commands, which can be directly applied to the standard proportional valves of the real machine. We test the controller on a 12-ton excavator in different types of soils. The experiments demonstrate that the controller can adapt online to changing conditions without the explicit knowledge of the soil parameters, solely from proprioceptive observations, which are easily measurable.