LEARN2SING: TARGET SPEAKER SINGING VOICE SYNTHESIS BY LEARNING FROM A SINGING TEACHER

Heyang Xue1, Shan Yang2, Yi Lei2, Lei Xie2, Xiulin Li3
Audio, Speech and Language Processing Group (ASLP@NPU), 1School of Software, 2 School of Computer Science, Northwestern Polytechnical University, Xian, China
3Databaker (Beijing) Technology Co., Ltd.

Abstract

Singing voice synthesis has been paid rising attention with the rapid development of speech synthesis area. In general, a studio-level singing corpus is usually necessary to produce a natural singing voice from lyrics and music-related transcription. However, such a corpus is difficult to collect since it’s hard for many of us to sing like a professional singer. In this paper, we propose an approach – Learn2Sing that only needs a singing teacher to generate the target speakers’ singing voice without their singing voice data. In our approach, a teacher’s singing corpus and speech from multiple target speakers are trained in a frame-level auto-regressive acoustic model and share the common speaker embedding and style tag embedding. Meanwhile, since there is no music-related transcription for the target speaker, we utilize log-scale fundamental frequency (LF0) as an auxiliary feature to be the inputs of the acoustic model for building a unified input representation. In order to enable the target speaker to sing without singing reference audio in the inference stage, the duration model and LF0 prediction model are also trained. Particularly, we employ domain adversarial training (DAT) in the acoustic model, which aims to enhance the singing performance of target speakers by disentangling style from acoustic features of singing and speaking data. Our experiments indicate that the proposed approach is capable of synthesizing singing voice for target speaker given only their normal speech samples.


arch


Examples of one singing teacher and two students


* Please ensure your web browser supports wav audio format.
* Student-1 is originated from an open-source corpus:https://www.data-baker.com/
* Real: real duration and LF0; Predict: predicted duration and LF0.


Target speaker Teacher Student-1 Student-2
Test condition  Original Synthesis Baseline Baseline+DAT Reference Baseline Baseline+DAT Reference
Real Predict Real Predict Real Predict Real Predict
Sample 1
Sample 2
Sample 3
Sample 4
Sample 5