View on GitHub

VoiceLoop

Voice Synthesis for in-the-Wild Speakers via a Phonological Loop
Yaniv Taigman, Lior Wolf, Adam Polyak, Eliya Nachmani

We present a new neural text to speech method that is able to transform text to speech in voices that are sampled in the wild. Unlike other text to speech systems, our solution is able to deal with unconstrained samples obtained from public speeches. The network architecture is simpler than those in the existing literature and is based on a novel shifting buffer working memory. The same buffer is used for estimating the attention, computing the output audio, and for updating the buffer itself. The input sentence is encoded using a context-free lookup table that contains one entry per character or phoneme. Lastly, the speakers are similarly represented by a short vector that can also be fitted to new speakers and variability in the generated speech is achieved by priming the buffer prior to generating the audio. Experimental results on two datasets demonstrate convincing multi-speaker and in-the-wild capabilities. In order to promote reproducibility, we release our source code and models.

Code: Link

Paper: Link

Network #1: Multiple speakers in-the-wild

Raw output before denoising:


Input: “Facebook Artificial Intelligence is working on this deep machine learning stuff”

Donald:
Mark:
Barack:
Sheryl:

Input: “I’m extremely happy and excited to officially announce my candidacy to the president of the Unites States”

Donald:
Mark:
Barack:
Sheryl:

Input: “I’m a neural network designed by Facebook Artificial Intelligence Research”

Donald:
Mark:
Barack:
Sheryl:

Input: “I’m looking for a job as a digitial assistant”

Donald:
Mark:
Barack:
Sheryl:

Priming for style

Input: “I’m looking for a job as a digitial assistant”

Donald A:
Donald B:
Donald C:
Donald D:

Input: “I’m extremely happy and excited to officially announce my candidacy to the president of the Unites States”

Donald A:
Donald B:
Donald C:
Donald D:

Network #2: Multiple speakers from VCTK:

Input: “Some have accepted this as a miracle without any physical explanation”