End-To-End Deep Multimodal Clustering

Deep multimodal clustering is challenging, since it needs to learn appropriate features for different modalities and find correct clusters by using consistency among the modalities. Existing methods treat the two problems separately, nevertheless, the optimization of one does not guarantee the optimization of the other. In this paper, we propose an end-to-end Deep Multimodal Clustering (DMMC) framework, which achieves a joint optimization of feature learning and multimodal clustering. We encourage the learned features of each modality to be cluster-oriented by autoencoder with Frobenius norm loss. Then, DMMC integrates features of different modalities by clustering integration layer and learns consensus feature by a consensus loss. The clustering result can be discovered in the consensus feature. Compared with existing clustering methods, our framework achieves superior performances measured by the standard clustering evaluation metrics.