Adaptable Weighted Voting Fusion for Multi-modality-based Action Recognition

In action recognition tasks, voting fusion can be used to combine classification results from multiple modalities to improve recognition accuracy and robustness. This paper proposes a novel weighted voting fusion method for multi-modality-based action recognition, which includes a weight generation method and three fusion strategies based on these weights. For weight generation, action instances are first classified based on single modality to obtain prediction scores for each action class. An adaptive weight for each modality is then generated by assigning a higher value to the modality with better classification, which is used for balancing the modality contributions of different actions. In the fusion stage, three fusion strategies are proposed to apply the adaptive weights to obtain the final class label, including maximum fusion, elimination weighted voting and maximum weighted voting. Experiments conducted on three kinds of modality fusion demonstrate the effectiveness of the proposed method.