Joint Analysis of Acoustic Scenes and Sound Events in Multitask Learning Based on Cross_MMoE Model and Class-Balanced Loss

Acoustic scene classification (ASC) and sound event detection (SED) are two research directions in the field of acoustics, and they are closely related. Previous works have adopted a joint analysis method for acoustic scenes and events based on multitask learning (MTL). However, the traditional MTL models are often sensitive to the proportion of dataset partitioning, and multitask analysis is not as effective as single-task analysis. In addition, the performance of traditional MTL models is highly dependent on the weights of the loss function, and manually adjusting weights is costly. In response to these issues, we suggest improvements in both the model and loss function formulation, to utilize additional sound event information to assist in improving the performance of ASC. First, the multigate mixture-of-experts (MMoEs) model is introduced into the field of acoustics. Experimental results obtained using TUT Sound Events 2016/2017 and TUT Acoustic Scenes 2016 datasets indicate that the mixture-of-experts model achieves an optimal performance of 98.74% in terms of
$F1$
-score, which is 1.43% higher than traditional MTL models; second, we improve the mixture-of-experts model and propose the Cross_MMoE model, which increases the information interaction between different task branches, and the
$F1$
-score is further improved to 99.04%; finally, to address the issue of imbalanced sample categories in the dataset, we evaluate the class balanced loss formulation to replace the traditional multitask loss function. The performance of the traditional multitask model, MMoE model, and Cross_MMoE model has been improved, and more specifically, the
$F1$
-score of the Cross_MMoE model has increased to 99.31%.