An Improved Frame-Unit-Selection Based Voice Conversion System Without Parallel Training Data

A frame-unit-selection based voice conversion system proposed earlier by us is revisited here to enhance its performance in both speech naturalness and speaker similarity. Speaker independent, bilingual (Mandarin Chinese and American English) deep neural net (DNN) acoustic model’s output, frame-level phone posterior probability (PPP), is used to represent the phonetic information. The corresponding frame-level F0 is used as the prosodic information. Kullback-Leibler divergence (KLD) between source and target PPPs (phonetic distortion) and the absolute difference between normalized source and target F0 (prosodic distortion) are used for selecting target frame candidates to construct a search lattice. The optimal target unit trajectory is obtained by Viterbi algorithm which tries to minimize the dynamic acoustic difference between the acoustic trajectory of the source speech and target candidates. The obtained spectral trajectory together with the enhanced pitch period and pitch correlation trajectory are sent to LPCNet vocoder to synthesize the converted waveforms. Compared with the top-rank system in Voice Conversion Challenge 2018, our new system can achieve on-par performance on studio to studio American English VC test, and better performance on non-studio to studio Mandarin Chinese VC test, in both speech naturalness MOS and speaker similarity DMOS.

1. INTRODUCTION

Voice conversion (VC) aims to modify the speech uttered by the source speaker to sound as if it was spoken by the target speaker without changing the linguistic information. The conventional voice conversion approach usually needs parallel data from source and target speakers. The parallel data is first aligned by dynamic time warping, then a mapping function can be trained to convert speech from source to target speaker. Several statistical conversion models such as Gaussian mixture model (GMM) [1], [2], neural networks [3], [4], [5], [6], non-negative matrix factorization [7], [8] have been proposed. Recently sequence-to-sequence modeling [9], [10] which adopts encoder-decoder architecture with attention has been studied for voice conversion task. This method can achieve appropriate duration conversion which is quite difficult in conventional methods.