Multimodal multi-label text classification based on bidirectional Transformer

Research on multi-label classification has developed from traditional single-modal analysis to complex classification using multiple modalities. However, single-modal input models are insufficient for effectively analyzing the increasingly rich multi-modal data on the Internet. Therefore, a ResNeXt-Bert method is proposed: introducing models for image and text information input to overcome the limitations of single-modal input data. The ResNeXt is used to extract features of images, and contextual embedding is input using early fusion method, and then fed into a pre-trained multi-layer bidirectional Transformer encoder model for classification. On the MM-IMDB dataset for studying multi-label classification problems, the micro F1 value reached 64.7%, which is better than other models. The experimental results show that introducing image and text information can indeed improve the multi-label text classification ability of the model.