Enabling Gesture-based Application Interaction on Head Mounted VR Display

With the integration of smartphones, VR head-mounted displays that are affordable (HMDs) have made VR more accessible to the general public and increased its widespread use. However, due to their lack of interaction, these systems frequently have limitations. GestOnHMD, a technique for a pipeline for classifying gestures and gesture-based interaction, is presented in this study. It makes use of the gesture technique in smartphones to detect hand gestures, detecting 21 points on hand on VR headset. Initially, gesture-elicitation research is carried out using specific headset the Google Cardboard to detect the 21 hand points which detects the hand gestures and display the meaning of the hand gesture. Based on signal detectability and user choices, different kinds of gestures are selected to input the dataset. It uses the neural networks to map the hand points and the cam detects the hand gestures and recognizes it and the displays the meaning of each On the VR device’s screen. Each gesture can be made specifically for the commands and its meaning by showing the hand gesture in front of the cam and storing it in the dataset. It can be used to read American Signal Language, etc. which makes people to understand hand gestures and might help in future exploring. The output is displayed on the Google Cardboard Virtual Reality device screen which is connected to the laptop.