Flood Detection in Social Media Using Multimodal Fusion on Multilingual Dataset

Social media has emerged as a information source for natural disaster detection and assessment, such as flooding. Although visual features from images and textual information have been used for this purpose before, research shows that fusion of these two modes can further improve the detection and assessment. Moreover, due to advancement of social media, users now can write in these social media using their native language. Therefore, it is important to design a methodology that can handle multiliguality, specially for global natural disasters assessment such as flooding. In addition, social media contents, such as image and text, contain dissimilar credibility based on the nature of the media. Here, we propose a fusion methodology to fuse visual features and textual features using a novel machine learning approach, and test it in multiple language datasets, to detect flooding in three different types of social media. The results show that the proposed methodology fuses the two modes efficiently and improves the classification performance in all experimented languages.

1 Introduction

Social media, such as Twitter and Flickr, have emerged as a source of rapid communication and information during natural disasters. However, the lack of reliability due to anonymity and uncertainty makes using social media for disaster assessment challenging. Therefore, different techniques including using visual features and textual features have been tested to improve the natural disaster detection in social media posts [19]. Research shows that the fusion of these two modes can improve the detection process even further [20]. However, it is important to analyse these two modes properly and employ a coherent fusion method for this purpose, otherwise the fusion can also hamper the detection and assessment process [18].