Interrelated Fusion CNN with Statistical Grouping among Multipatches for Occluded Facial Expression Recognition

Facial expression recognition in-the-wild is still a challenge due to unpredictable occlusion. In this paper, we propose an Interrelated Fusion CNN (IRF-CNN) with more attention on the non-occluded facial regions for expression recognition. With a new patch masking scheme, facial landmark patches and their corresponding contextual patches are obtained. A self-attentive mechanism assigns the patches with some attentive weights, which reflects their importance for expression recognition task. Two groups of patches corresponding to high and low weights respectively are obtained by analogous pooling with statistical indicators of their attentive weights. The adaptive grouping scheme could assign the key non-occluded patches for facial expression into the high-weight groups, especially in the case of unbalanced weight distribution. An interrelated fusion module with an attention stacking strategy is proposed for integrating complementary and reinforcing representation of the two groups of patches. IRFCNN is evaluated on in-the-lab facial expression datasets (CK+, Jaffe) and in-the-wild facial expression datasets (SFEW, RAF, FED-RO). Experiment results validate that the proposed method has a competitive performance.