Beginner's question reg data sets needed for fine-tuning

[georgedei]

What is val_list and OOD_list used for and how should I create them? I want to add a single English speaker with specific accent to LibiriTS. Can I use the exisiting LibriTS val_list and OOD_list or should I create ones that use the new speaker only? I don't have much material for the new speaker, so modeling from scratch is not an option.

[IIEleven11]

What is val_list and OOD_list used for and how should I create them? I want to add a single English speaker with specific accent to LibiriTS. Can I use the exisiting LibriTS val_list and OOD_list or should I create ones that use the new speaker only? I don't have much material for the new speaker, so modeling from scratch is not an option.

I have a fine tuning guide you can look at that directs you through the creation of those. There is a link to it on the repo's readme.

The validation list or val_list is a portion of your dataset that the model will use after each epoch to evaluate it's current performance. It compares the predictions on the validation dataset with the actual values. This can help you gauge how training is going and if anything needs to be adjusted.

OOD_list works in almost the same way except it's data the model has not seen or data that was not like training data. This is important because it tests the models ability to generalize it's learning on new unseen data.

[georgedei]

Thanks 1111, gradually I'm getting some ground under my feet.

[georgedei]

Thanks to your help, finally got the finetuning to work. On a 4090 card, batch size 2, max_length 280, about 1 hour and 900 clips of data.
Looks like my system has CPU bottleneck, since the 4090 was running at 50% average, but 40 epochs got done in about 8 hours.
It seems I can rent a Google VM with A100, that would allow to run with max_length 800.
Is the quality difference siginificant?

[georgedei]

Question about OOD_list: Looking at the OOD_texts in the data directory, it has 141,000 entries and is actually a copy of train_list_libritts.txt. Could it be that the libritts texts were used for training of LJSpeech? The audio files listed there are not present - does it mean that OOD_list contains only text, and the audio files it contains are being ignored? Do I really need 141,000 entries for fine tuning? I also noticed some entries are far longer than 10 seconds. It seems to me, since the OOD texts were taken from the libritts training, its not really out-of-distribution, does it make sense to use it for fine tuning libritts model? Would you have some more details on construction the OOD list for fine tuning. Thank you.

[IIEleven11]

OOD_list.txt is just a set of data to reference against so we can decide how to adjust any parameters during the tuning process. There is no accompanying audio with OOD_list.txt in this instance. And you're right the list does come from libriTTS. I haven't looked into exactly how they used it though. There are quite a few known issues with that data set. You can in theory make your own OOD_list.txt, i've played around with this with varying results. I'm unsure how large to make it, but just make sure you phonemize and properly punctuate. You should check out https://github.com/IIEleven11/StyleTTS2FineTune and scroll down a bit to where I touch on OOD_list.txt also @Kreevoz made a very detailed set of notes. #81

[Kreevoz]

@georgedei If you want to make your own file, you should make your OOD_text extremely diverse. It does not have to be enormously large - mine only contains 2000 lines of text. Since the file won't be sampled that often during adversarial training, that is sufficient.
Since libriTTS is not exactly diverse enough (and it has quality issues), I made a new file that contains some tech articles, some scientific articles with really complicated words, some passages from books, some lines of dialog from games, etc. All broken up into medium to short sentences, or sentence fragments if they were too long.

[georgedei]

just one more question about the format of OOD_text, is this correct, or do I need to include a dummy file name:
text |
or
whateverfilename | text |

[IIEleven11]

I would just mirror the exact format within OOD_list.txt. if I remember correctly it's
audio1.wav|transcription|transcription. Maybe

[Kreevoz]

@georgedei There is a conditional check in the OOD dataloader which checks for the type of formatting.
If it finds .wav in the first field (the filename), then it will assume the format is: irrelevantfilename.wav|transcription|speaker_id
The filename can be fictional, it does not need any audio files to really exist.

If it finds no .wav in the first field, it will assume the format is transcription | speaker_id , and take the transcript from the first field.
So you can use two variants.

I personally use the second transcription|0 version since it's easier to assemble.