Multimodal Feature Enhancement Plugin for Action Recognition

The advanced action recognition technologies have attempted to adopt multimodal architecture based on powerful pretrained models (e.g., visual-text) to achieve higher accuracy. However, visual encoders treat all frames equally, which may lead to (i) loss of key information and (ii) poor robustness, as different video frames may contribute differently to the visual representation. The preliminary experiments show that the assessment of frame inherent features could be highly correlated with the classification accuracy of the visual encoder. Based on this key insight, this paper proposes MEP (Multimodal-feature Enhanced Plugin), a frame selection method based on visual inherent features to guide the model in selecting video frames, thereby enhancing the accuracy of multimodal action recognition. Specifically, MEP first leverages a pretrained text encoder and a pretrained visual encoder to extract different modality features. For the pretrained visual encoder, we design a plug-and-play module for frame selection, which can use visual features as well as clustering methods to achieve efficient frame selection. Additionally, we introduce a module to extract textual inherent feature (i.e., emotional features) to enhance text representation. Experimental results show that the proposed method achieves performance comparable to or better than existing state-of-the-art video action recognition models on multiple datasets.