Working well, but get odd voice output at some end quotes.

[bmcgonag]

Drew, this is working really well thus far. I do, however, get some odd sounds at a few end quotes. I've looked, and tried to remove extra spaces, and make sure all the quotes are proper end quotes. Not sure why it addes in these odd noises from the reader voice in these places. I've seen this with both .txt and .epub files. Any ideas?

[DrewThomasson]

It's probably hallucinations from the xtts model,

It is a transformer based model so

I suppose you could like lower its temperature on inference?

But I've never dived that deep into using xtts to be honest.

Also you could try using STYLETTS2 model instead, it's not quite as good as XTTS but I don't think it has the hallucination issues

[bmcgonag]

Ok, let me see what I can figure out.

[KingKestrel]

I've seen this too. xtts works beautifully with non fiction books since there's so little quotes, but in fiction hallucinations happen at the end of most quotes. I kind of wonder if xtts is struggling to pivot between what it interprets as different speakers and the narrator.

Drew what you've done here is awesome. Really appreciate the tool and I use it quite a lot. Thank you for your work!

[DrewThomasson]

Actually I think I've solved this in my ebook2audiobookxtts repo:

• You just need to turn the temperature down from the default setting of .65 for xtts

• That'll reduce the hallucinations but make it less creative with tone and such

It'll be adding temperature=.65(or whatever lower value, setting it to even only .1 doesn't seem to cause any issues.)

To all lines that have this

tts.tts_to_file(

I'll probs add this into the new gradio version of VoxNovel I'm working on (once I get it up and working lol)

More info can be found here ⬇️

COQUI XTTS DOCS

[sotir3d]

I tried your fix of changing the temperature. As you said, it did improve hallucinations, but did not seem to fix the issue entirely.
From my testing, hallucinations seem to appear when a paragraph ends on a quotation mark. The model almost always hallucinates when a paragraph ends with ?" or !" (so a 'spoken' question or exclamation with an ending quote.). Paragraphs not ending on a quote (almost?) never seem to cause hallucinations, while paragraphs ending on ." sometimes cause them.

In your 'mini_story - Drew.epub' for example, it will 100% hallucinate for me at the sentence '“Moths? My dear, these silks could turn away the sharpest of blades!”', no matter what sampler settings I try.

All my testing was done using ebook2audiobookxtts.

I am unsure if this is an inherent limitation of XTTS, but if this could somehow be fixed, this would make an already great project absolutely fantastic imo, since XTTS seems to be the by far most natural sounding TTS model for audiobook narration.

[DrewThomasson]

Wait ...

Did you set the temperature to 1 or .1

Cause 1 is really high lol

You should be lowering it to (0.1) for less hallucinations

[sotir3d]

I tried both. 0.1 was actually horrible for me, but I only tried one gen with it. 1 came out almost perfect though, apart from the stuff I mentioned (a 4 hour gen with around 40k words.)

[DrewThomasson]

@sotir3d
Moving this discussion over here

to the ebook2audiobookxtts repo issues cause this is now talking about that lol

DrewThomasson/ebook2audiobook#34 (comment)