Multimodal Dual Cross-Attention Fusion Strategy for Autonomous Garbage Classification System

Existing garbage classification methods based on pure images are difficult to distinguish garbage with similar appearance but different materials, and have problems of low accuracy and poor robustness, making them difficult to promote and apply in practice. Inspired by the combination of “looking” and “listening” when people distinguish objects, we propose a method that combines image with impact sound for garbage classification. However, due to the lack of homogeneity of image and audio, how to effectively fuse these two types of information is a key challenge to improve the accuracy and robustness of garbage classification. In this article, we propose a multimodal dual cross-attention fusion strategy (MDCF strategy) to fuse multisource information. This strategy includes a method that combines image and audio to solve the challenge existed in pure image classification methods and a fusion strategy based on dual cross attention, which takes into account both the intramodal relationship and the intermodal relationship. Through our self-built multimodal dataset, we demonstrated that the MDCF strategy has a high recognition rate, excellent robustness, and fast processing time. The MDCF strategy was deployed into a garbage classification device that we designed to establish an autonomous garbage classification system. Practical application experiments show that our system has industrial-grade application capabilities, high recognition rate, fast identification time, and excellent robustness. Our strategy not only opens a door for large- scale application of source garbage classification, but also provides a general framework for the fusion of multisource information, which can be extended to various information fusions.