An sEMG-Controlled Prosthetic Hand Featuring a Tiny CNN-Transformer Model and Force Feedback

This paper introduces the design of a wireless prosthetic hand with surface electromyography (sEMG)-based control and force feedback. The system features a novel CNN-Transformer model for classifying 21 gestures from sEMG signals. With a tiny size of only 169 kB, the proposed CNN-Transformer model achieves a high average accuracy of 81.39% when evaluated on a public dataset across 10 subjects. The system integrates wireless electronics into a 3D-printed open-source robotic hand. A custom-designed flexible sEMG armband is used for signal acquisition and stimulation delivery. A mobile device is used for real-time model inference. By employing hardware machine learning accelerators on the mobile device, a short inference time of 0.375 ms was achieved, which is 3.2x faster compared to using the CPU alone, rendering a smooth user experience with low latency. Force sensors were mounted on the robotic hand’s fingertips, and the outputs were modulated and delivered as electrical stimulation through the surface electrodes to provide force feedback to the user. The developed deep learning model and system design approaches have broad applicability across a wide range of prosthetic applications.