Touchless Control of Heavy Equipment Using Low-Cost Hand Gesture Recognition

Human-machine interaction using remote hand gestures is becoming increasingly prevalent across various industries. However, their potential application to heavy construction equipment is often overlooked. This article presents a robust and inexpensive hand gesture recognition system that was implemented and tested on a robotic 1-tonne wheel loader. The system uses an RGB camera paired with a laptop to process, in real time, hand gestures to control the loader. We first design four unique gestures for controlling the loader and then collect 26,000 images to train and test a neural network for hand gesture recognition. Our system uses robust landmark detection using an off-the-shelf system prior to gesture recognition. We successfully controlled the loader to excavate in a rock pile by using the proposed hand gesture recognition system.