A deep reinforcement learning based approach towards generating human walking behavior with a neuromuscular model

A gait model capable of generating human-like walking behavior at both the kinematic and the muscular level can be a very useful framework for developing control schemes for humanoids and wearable robots such as exoskeletons and prostheses. In this work we demonstrated the feasibility of using deep reinforcement learning based approach for neuromuscular gait modelling. A lower limb gait model consists of seven segments, fourteen degrees of freedom, and twenty two Hill-type muscles was built to capture human leg dynamics and the characteristics of muscle properties. We implemented the proximal policy optimization algorithm to learn the sensory-motor mappings (control policy) and generate human-like walking behavior for the model. Human motion capture data, muscle activation patterns and metabolic cost estimation were included in the reward function for training. The results show that the model can closely reproduce the human kinematics and ground reaction forces during walking. It is capable of generating human walking behavior in a speed range from 0.6 m/s to 1.2 m/s. It is also able to withstand unexpected hip torque perturbations during walking. We further explored the advantages of using the neuromuscular based model over the ideal joint torque based model. We observed that the neuromuscular model is more sample efficient compared to the torque model.

I. Introduction

The gait model capable of reproducing human-like locomotion can help us further understand the human locomotion control scheme which can be used for developing bipedal robots and wearable robots (e.g. exoskeletons, prostheses, etc.). For instance, with a simple inverted pendulum model, it has been shown that the human-like bipedal walking gait can be achieved passively (without active control) because of the natural dynamics of the human body [1]. It has also been found that both human walking and running gait can be described by a simple spring loaded inverted pendulum model [2], [3]. Several legged robots were developed and successfully demonstrated the control benefits of these models [1], [4], [5]. Besides, the bio-inspired conceptual model also shows benefits for the exoskeleton control [6].