Intent transfer in speech-to-speech machine translation

This paper presents an approach for transfer of speaker intent in speech-to-speech machine translation (S2SMT). Specifically, we describe techniques to retain the prominence patterns of the source language utterance through the translation pipeline and impose this information during speech synthesis in the target language. We first present an analysis of word focus across languages to motivate the problem of transfer. We then propose an approach for training an appropriate transfer function for intonation on a parallel speech corpus in the two languages within which the translation is carried out. We present our analysis and experiments on English↔Portuguese and English↔German language pairs and evaluate the proposed transformation techniques through objective measures.

1. INTRODUCTION

Intent in speech is manifested in how a sentence is delivered, its phrasing, rhythm, intonation, energy and voice quality. Given the same sentence, speakers have large variability and freedom in focussing any concept (word) they choose to, and the degree to which the emphasis is laid. This prominence pattern of the words in an utterance bears information about the words' relevance, given/newness etc., in addition to the general style of the speaker. These aspects broadly fall under the ‘augmentative’ and ‘affective’ parts of prosody, the extra information in speech, to ensure that the intended message is unambiguously decoded by the listeners [1]. In this work, we use the term ‘intent’ to exclusively refer to such aspects within intonation. While it is ideal for Text-to-speech (TTS) systems to synthesize as appropriate to the underlying meaning of the sentence, intent is largely under-represented in text. However, there are certain domains where this information may be accessible to the synthesizer. In this work, we deal with one such domain, speech-to-speech machine translation (S2SMT). The goal of S2SMT is to take as input, speech in one language and automatically ‘dub’ it to generate as out-put, a translated sentence with the same meaning spoken in another language.