Voice Cloning: When to Fine-Tune Pretrained TTS Models and How Much Data is Needed?

[ClaudiuFilip110]

Hi, I'm a ML Engineer (with a few years experience, but) new to TTS (and audio ML). I have experience primarily with NLP and LLMs, but I’m working on a Voice Cloning project, and the transition into TTS has been a bit confusing.

Here's my plan, if you have any tips or suggestions please feel free to add them here.

English Voice Cloning

1. I'm starting with pre-trained voice cloning models (xTTS, VITS, etc), but I want to improve their performance. Do I use RVC to do that or do I go straight to fine-tuning?
2. If I need to fine-tune the model to better match a particular voice, do I need a dataset specific to each user? In other words, will I need to create a separate dataset for each user, or can a single dataset achieve better results?
3. How much training data should I have per user for fine-tuning?

Another language Voice Cloning
I'm planning to train my own model from scratch (or from a checkpoint).

• create a dataset of similar size to LJSpeech (24hr audio)
• have multiple speakers instead of 1
• train the model from VITS or xTTS checkpoints

P.S. Any tips are welcome, as I said, I'm quite the novice when it comes to anything audio ML-related.

[eginhard]

1. You can try both
2. Just fine-tune with all of the data combined
3. Depends on the data, you'll need to try

[madey83]

@eginhard

is it mean that i can use VITS preprained polish male voice and based on it use my LJSpeech dataset with my voice to train new model?

[eginhard]

Sure

[pivolan]

As my expirience, to train from the scratch on dutch, de and flemish laguages, 20 hours is enough. But 50 hours got really good quality. For finetuning you need at least 2 hours, but you will hear some corrupts comprimised quality. 6 hours is good enough for usage.