Self-Attention With Restricted Time Context And Resolution In Dnn Speech Enhancement

The multi-head attention mechanism, which has been successfully applied in, e.g., machine translation and ASR, was also found to be a promising approach for temporal modeling in speech enhancement DNNs. Since speech enhancement can be expected to take less profit from long-term temporal context than machine translation or ASR, we propose to employ self-attention with modified context access. We first show that a restriction of the employed temporal context in the self-attention layers of a CNN-based network architecture is crucial for good speech enhancement performance. Furthermore, we propose to combine restricted attention with a subsampled attention variant that considers long-term context with a lower temporal resolution, which helps to effectively consider both long- and short-term context. We show that our proposed attention-based network outperforms similar networks using RNNs for temporal modeling as well as a strong reference method using unrestricted attention.