Distinctive and Natural Speaker Anonymization via Singular Value Transformation-Assisted Matrix

Speaker anonymization is an effective privacy protection solution that aims to conceal speaker's identity while preserving the naturalness and distinctiveness of the original speech. Mainstream approaches use an utterance-level vector from a pre-trained automatic speaker verification (ASV) model to represent speaker identity, which is then averaged or modified for anonymization. However, these systems suffer from deterioration in the
naturalness
of anonymized speech, degradation in speaker
distinctiveness
, and severe privacy leakage against powerful attackers. To address these issues and especially generate more natural and distinctive anonymized speech, we propose a novel speaker anonymization approach that models a matrix related to speaker identity and transforms it into an anonymized singular value transformation-assisted matrix to conceal the original speaker identity. Our approach extracts frame-level speaker vectors from a pre-trained ASV model and employs an attention mechanism to create a speaker-score matrix and speaker-related tokens. Notably, the speaker-score matrix acts as the weight for the corresponding speaker-related token, representing the speaker's identity. The singular value transformation-assisted matrix is generated through the recomposition of the decomposed orthonormal eigenvectors matrix and non-linear transformed singular through Singular Value Decomposition (SVD). This process prevents the degradation of speaker distinctiveness caused by the introduction of other speakers' identity information. By multiplying the singular value transformation-assisted matrix and speaker-related tokens, we generate the anonymized speaker identity representation, thereby producing anonymized speech that is both natural and distinctive. Experiments on VoicePrivacy Challenge datasets demonstrate the effectiveness of our approach in protecting speaker privacy under all attack scenarios while maintaining speech naturalness and distinctiveness.