Hand Action Recognition from RGB-D Egocentric Videos in Substations Operations and Maintenance

This paper proposes a novel multimodal fusion network (MRDFNet) for egocentric hand action recognition from RGB-D videos. First, we utilize three separate streams to extract individual spatio-temporal features for different modalities, which include RGB frames, optical flow stacks, and depth frames. Particularly, for RGB and depth streams, an Attention-based Bidirectional Long Short-Term Memory network (Bi-LSTA) is used to identify regions of interest both spatially and temporally. Then, the extracted features are fed into a fusion module to obtain the integrated feature, which is finally used for egocentric hand action recognition. The fusion module is capable of learning complementary information from multiple modalities, i.e., preserving the distinctive property for each modality and meanwhile exploring the shareable property across modalities. Experimental results on both self-collected RGB-D Egocentric Manual Operation Dataset in Electrical Substations (REMOD-ES) and the THU-READ containing daily-life actions show the superiority of the proposed approach over state-of-the-art methods.

I. Introduction

Regular maintenance is essential for electrical systems to avert system and equipment malfunctions. It also guarantees optimal safety and efficiency when utilizing facilities. Electricians and maintenance personnel with expertise in maintenance procedures usually handle the upkeep of electrical equipment. They commonly adhere to an electrical maintenance checklist. To ensure the safety of the work environment when working with or around electrical equipment, personnel must consistently observe electrical safety protocols within their designated areas. These measures encompass utilizing appropriate safety gear, deactivating power sources, examining electrical equipment for any signs of damage, and executing operations meticulously and reliably. Except the safety consciousness of the personnel, strengthening the management of power safety production can also ensure the stability and safety of the power system. With the development of intelligent surveillance technology, such as action recognition, timely identifying unsafe actions of the personnel and providing reminders or alerts to prompt correction could greatly contribute to the electrical systems.