Method for Recognizing Hand Gestures in Real- Time Across Multiple Sensors Using Machine Learning

The field of human-computer interaction includes multimodal gesture detection as a key task. In human-machine interface, wearable sensors for hand gestures consume additional potential than ambient detection due to their short price, weightiness, and wide applicability. Despite the fact that research on hand motions with wearable sensors already exists, the results are rarely transferable from one scenario to another. And when it comes to collaboration between humans and technology, there is still no agreed-upon protocol for hand signals. In this study, we introduce a new technique of using hand gestures to command the environment. An affordable Internet of Things (IoT) sensor-pluggable device that has an integrated accelerometer and gyroscope is used to collect the data for analysis. Inputs to the feature extractor network include the raw data from the RGB sense modality, the visual flow sense modality, and the disassembled information from the hands. This work provides an additional modality, feature extraction based on temporal differences in RGB sense modality. The characteristics of these modes are learned using a pre-trained version of the Googlenet Caffe model. All of these components need to come together as a whole for the fusion procedure to be successful. When doing a classification, the bidirectional LSTM network is used. The new approach is put through its paces using the IPN Hand dataset. When first starting out, each modality is employed independently. Then, we employ such mixtures to boost classification precision even further. Combining the segmented indicator sense modality with the optical movement mode and the chronological transformation of the RGB sense modality yielded the greatest results. The suggested method for multimodal gesture recognition outperforms the benchmark methods, as shown by the simulation results