A Disentangled Recurrent Variational Autoencoder for Speech Enhancement

The variational autoencoder (VAE) is a powerful generative model of speech signal. In the problem of speech enhancement, the VAE is used to model the clean speech whereas the non-negative matrix factorization is used to model the noise. The Wiener filter can be applied to extract the clean speech from the mixture signal. In this paper, we extend this approach by proposing a more sophisticated VAE framework, so-called disentangled recurrent variational autoencoder (RVAE), to model the clean speech. It takes the strategy of pre-training plus fine-tuning. Firstly, the clean speech and noise are pre-trained using two dependent RVAEs. Secondly, by transferring the knowledge learned in these two RVAEs, the third RVAE is used to train the mixture signal, resulting in disentanglement of the latent variables of the clean speech and noise. Experimental results demonstrate that the proposed approach outperforms other VAE-based approaches in terms of the signal-to-distortion ratio.