DEP-SNN-RL:Spiking Neural Networks Reinforcement Learning in Musculoskeletal Systems

Muscle-actuated organisms exhibit an extraordinary ability to learn a wide array of agile movements. However, replicating such versatility and efficiency in reinforcement learning (RL) poses significant challenges, primarily due to the complexity of over-actuated action spaces. These challenges are often attributed to the sample efficiency issues prevalent in RL, compounded by the inefficacy in exploration strategies within such expansive action domains. To address the challenge of ineffective exploration in over-actuated spaces, we leverage Differential Extrinsic Plasticity (DEP), an innovative self-organizing mechanism designed to enhance and expedite exploration across the state space. To further augment the sample efficiency of reinforcement learning techniques, we introduce an integration with the third generation of neural networks, namely Spiking Neural Networks (SNNs). This integration, forming the core of our DEP-RL framework, sets a new benchmark for rapid and effective learning in musculoskeletal systems. Our approach not only surpasses the performance of conventional DEP-RL methodologies but also marks a significant leap forward in advancing reinforcement learning capabilities within complex, muscle-driven biological architectures.

I. Introduction

The study of musculoskeletal models as overactuated systems, where the number of muscles significantly surpasses the degrees of freedom, poses a fascinating research challenge [1], [2]. These models demonstrate exceptional performance, highlighting the complexity and efficiency of biological systems in executing movements. Reinforcement learning, inspired by biological behaviors [3], offers a compelling approach for exploring decision-making processes within such overactuated systems. It holds the potential to enhance our understanding of how these models achieve their remarkable capabilities [4]. Nonetheless, existing reinforcement learning techniques have shown limited success in addressing the decision-making challenges inherent to these complex problems. This gap underscores the need for advanced methodologies that can more effectively capture the intricacies of musculoskeletal systems and improve decision-making performance in overactuated models.