Leveraging Ultrasound Sensing for Virtual Object Manipulation in Immersive Environments

Hand gesture recognition is a fundamental component of intuitive and immersive user interfaces in virtual reality (VR) applications. This paper presents a data-driven approach utilizing ultrasound data and deep learning techniques for hand gesture recognition in VR interfaces. The proposed methodology involves acquiring data from a subject, training a model using the acquired data, and evaluating the model’s performance on both the training data and during real-time inference. The evaluation metrics primarily focus on accuracy percentage: measuring the classifier’s performance in correctly classifying hand gestures. 4 hand gestures were primarily considered for the study and demonstration. For offline evaluation with a 20% test-train split, an accuracy percentage of 91% was observed. For online evaluation, an accuracy percentage of 92% was achieved. Results on the classification of 7 hand gestures were also analyzed for both online and online evaluation due to the promising results from the 4 gesture classification. The latency of the pipeline, from ultrasound data acquisition using screenshots to sending commands for VR object manipulation, was measured to be 59.48 milliseconds. The results demonstrate the effectiveness of the approach in accurately recognizing hand gestures, both during training and in real-time inference. We supplement our results with a video of the forearm ultrasound data being used to control a custom-designed VR game in a low-latency fashion. This research provides valuable insights into the performance and applicability of ultrasound-based hand gesture recognition techniques in VR interfaces. By employing deep learning and leveraging real-time data acquisition, this approach paves the way for intuitive and immersive interactions in various VR applications. The study contributes to the field of body sensor networks, highlighting the potential of forearm ultrasound based data-driven techniques for enhancing user interaction and immersion in VR environments.