DetTrans: A Lightweight Framework to Detect and Translate Noisy Inputs Simultaneously

Neural machine translation (NMT) systems trained on clean data usually suffer from performance degradation when translating noisy inputs. Existing works attempt to improve the robustness of NMT normally via data augmentation, where synthetic noisy data are mixed with original clean data, either for training NMT with the standard NMT loss alone, or for tuning auxiliary tasks in a multi-task learning manner. Typical auxiliary tasks include detecting and correcting noises, exploiting noisy outputs for contrastive learning etc. The aforementioned two auxiliary tasks are generally designed independently, and the modules for detecting and correcting noises are heavyweight. In this article, we propose a new framework, DetTransNet (Detector-Translator Network), aiming to detect positions of noises in the input and translate the input simultaneously. The newly introduced noise detector module is essentially a lightweight binary classifier built upon the final layer of the encoder of the original Transformer model for the translation task, which is to identify at which position of the input has potential noise. The module has a very few parameters. In order to help the model capture the relationship between clean instances and their noisy counterparts, an extra loss is further introduced to enhance the interaction between clean and noisy data. In this way, we combine noise detection and contrastive learning together. As the model is able to identify and locate noises, a heuristic method is proposed to correct detected noises, in order to achieve better translations. Experiments show that DetTransNet is robust to four types of noises (deletion, insertion, swapping, keyboard), and obtain a substantial improvement of up to 1.6 BLEU points across different datasets.

I. Introduction

Neural machine translation systems [1] are traditionally trained by textual instances. Those instances usually need to go through additional pre-defined heuristic data processing pipelines to be canonical for training MT, as described in the previous technical reports of the WMT shared tasks [2], [3], [4]. However, with the rapid development of 5G technology and the maturity of the mobile Internet industry, the data sources of machine translation systems have become more diverse: results generated by optical character recognition (OCR) or automatic speech recognition (ASR) could be input texts for NMT, where a wide variety of noises would be introduced. Moreover, users may type queries quickly on the mobile phones, with different kinds of misspellings. In summary, in real-world machine translation application scenarios, noisy inputs could be more common and cannot be ignored. Unfortunately, current research [5] has empirically shown that nerual machine translation systems are vulnerable to both synthetic and natural noisy inputs. Therefore, improving the robustness of translation is crucial for deployed NMT systems.