Hate Speech Detection using Multimodal Meme Analysis

Hateful memes, which often contain offensive language and derogatory images, have become increasingly prevalent on social media platforms, posing a significant challenge to content moderation efforts. To address this issue, the proposed approach explores the detection of hateful memes through a multimodal analysis that utilizes both visual and textual modalities. A novel lightweight architecture is proposed which involves generating object tags and image captions and utilising them to combine the visual and textual cues. In the final classification phase, various classification models, such as Support Vector Classifier and Gradient Boosted Decision Trees, are trained. This layer combines the results of concurrent processing branches (textual and visual) to provide predictions for the sample.