Localization-Driven Speech Enhancement in Noisy Multi-Speaker Hospital Environments Using Deep Learning and Meta Learning

This work addresses the problem of 3D-localizing and enhancing the speech of one main speaker in noisy multi-speaker hospital environments using a multi-channel microphone array. In our model, we propose conducting speaker localization using a machine learning model based on convolutional recurrent neural networks (CRNN) followed by minimum variance distortionless response (MVDR) beamforming. In addition, to ensure that our speech enhancement module is adaptive when deployed in different environments, we trained a meta learning model. Firstly, in the localization step, an estimation of the direction of arrival (DOA) in the elevation and azimuth planes is executed. This is conducted in a 3D space with the presence of noise, reverberation, and up to two more speakers. Using estimated DOA, the MVDR beamformer then enhances the speech of the main speaker. In order to test our model, we adopted and simulated a real-world problem where the objective was to enhance the speech of a clinician in a noisy intensive care unit (ICU) with the presence of other speakers. Furthermore, in order to validate our model, we adopted a speech-to-text module to evaluate the word error rate. Moreover, we implemented our algorithm on hardware using commercially available components and tested it in a real environment. Results showed that our model outperforms other machine learning and non-machine learning algorithms. Finally, we used our trained Meta Learning model to show that our model can adapt to new environments while maintaining high performance after retraining with only a few-shot recordings.