A Novel Approach to Improve User Experience of Mouse Control using CNN Based Hand Gesture Recognition

This paper presents a method that controls a mouse pointer using hand gestures and further improves the user experience by resolving the issue of mouse displacement due to abrupt hand movement. The approach is capable of detecting a set of predefined gestures such as clicking, scrolling, and dragging, allowing for a more natural and intuitive interaction with the computer. This research includes the study and use of the Mediapipe library, which is used to track the hand. The Mediapipe hand detection model is based on a machine learning algorithm that uses a variant of a SSD architecture, which is trained using a large dataset of hand images. The training data was annotated with bounding boxes that indicate the location of the hands in the images. The model was trained to predict the bounding boxes for the hands in new images using a combination of CNNs and other DL techniques. Using this library, many methods were developed to control the mouse using gestures. However, it has been observed that in existing systems, if hand positions change abruptly, it causes pointer displacement that negatively affects the user experience. The proposed system resolves this issue of mouse pointer displacement to further enhance the user experience in a variety of use cases that require abrupt and natural hand movements during the course of action. The proposed system solves the issue of mouse displacement by tracking the coordinates of hand movement and calculating the new position of the mouse pointer using the distance moved by the hand, rather than just using the position of the hand with respect to the screen. Using this method, the user can also control the mouse using a small part of the camera's wide angle to map it on any point on the screen. Experimental results show that the proposed solution is effective and efficient in terms of user experience in controlling the mouse pointer and holds potential for wider use in a variety of applications of human-computer interaction.