Bark's Future?

[DarthKX]

I have a question, I haven't installed your fork of Bark myself, but I'm interested in it, especially now that Suno and Udio are under fire from the majors... I use Udio to create some really great AI Metal, and it would piss me off to no end if the majors and the RIAA came and removed that ability so that they can be the only beneficiary of the tech.

I think opensource can save this mess, so I wanted to know if you have any intention of continuing and maybe adapting your fork to make sure that if they do neutralize Udio, mostly (it's 10x better than Suno though I started on Suno), that we can still generate some really great AI music for ourselves at least.

Also, and it's okay if you don't have time to answer this one, but how hard is it to train our own models of bark? I'd really like the ability to fine-tune the model to my needs, maybe train it with the stuff I've made on Udio for one. Is it complicated? Can this be done with a RTX 3090 (or a pair of them)?

With the tools progressing at the speed they do, I'm looking foward to one day manage to make full cinematic movies, and for that a good audio generator will also be needed, even if Sora eventually gets released, or someone else beats them to it. But a good movie also requires music. Obviously there are other ways of approaching this dream, but to do everything on my own, it's only possible with AI. By the way if anyone knows a really good video generator on git that can run locally, something in the veins of Krea, Runway ML3 or even Sora.

And last question, do you think that protrait animator thingy that anyone talks about on YT (Live protrait something), can be of use to improve Roop-unleashed? Cause I gotta say, if I can take my own shots of "acting" and have it applied this well in terms of facial animation, it would really help for said movie aspirations I have with AI tools.

Thanks in advance.
Keep up the excellent work
Cheers

[C0untFloyd]

I think opensource can save this mess, so I wanted to know if you have any intention of continuing and maybe adapting your fork to make sure that if they do neutralize Udio, mostly (it's 10x better than Suno though I started on Suno), that we can still generate some really great AI music for ourselves at least.

I'm sorry I very much lost interest in continuing my bark fork because I only have so much time, doing this in my spare time and roop already takes away a lot of my time. I already thought of having it archived to not create any false hopes. Also, Suno may be based on Bark techniques but Bark can hardly generate any music, if it does it is purely random.

how hard is it to train our own models of bark? I'd really like the ability to fine-tune the model to my needs, maybe train it with the stuff I've made on Udio for one. Is it complicated? Can this be done with a RTX 3090 (or a pair of them)?

Again, Bark is not about music, it is about generating emotional speech. Training is neither hard nor do you need expensive hardware. I finetuned german language using my RTX 2060 Super. The language you want to train needs to be already known by Bark. Training would take on average 6 hours I guess.

And last question, do you think that protrait animator thingy that anyone talks about on YT (Live protrait something), can be of use to improve Roop-unleashed?

You're talking about this https://huggingface.co/spaces/KwaiVGI/LivePortrait ?
It's very cool stuff, however I don't have any idea how this could be of use for face swapping. Would make a nice gimmick for the extras tab perhaps.
Oh perhaps you mean for livecam-swapping? Researched some more and stumbled over this:
https://github.com/warmshao/FasterLivePortrait
It seems to achieve realtime motions already. If this has matured some more, it really might be used for livecam swapping.

[DarthKX]

the first one... couldn't it be useful to apply the source video expression back to the post-processed image to make the expression less-wooden? I dunno, maybe it's complicated.

Fair enough on Bark not being for music, since it had that name I assumed it could do the same as Suno-Bark. What about singing though?

And please don't archive it, emotional speech is useful, I could try and use it for my audiobooks. Is it possible to save voices and always get the same voice actor back with it? Cause that would be a bit like 11labs... they are a bit expensive for my taste since I have over 20 books to make lol. I'm sure there will be other tools.

Let me know and thanks again for the great work. And I bet roop already takes a lot of your time. But then you'lol get full credit when my movie goes toe to toe with hollywood blockbusters ha ha. We're still ways off, maybe Sora will fix that, but let's see if it releases first.

I'll check the one for the portraits... I really am camera shy, so if I could put my book's hero's face on my video and it look real, that wouldn't be the worse thing :D.

Hopefully someone will tackle something like Udio, I would really be pissed if this was removed and I couldn't make AI Metal anymore. I love metal bands and artists, in fact I've helped them sell tens of thousands of album for over a decade as a journalist, but I make stuff I find better than my favorite bands now (I may be biased) and I want to keep doing it!

Cheers

[C0untFloyd]

couldn't it be useful to apply the source video expression back to the post-processed image to make the expression less-wooden?

In the future yes perhaps, currently no. If you tried it out you might have noticed it doesn't work with angles and source and target need to be aligned.

Fair enough on Bark not being for music, since it had that name I assumed it could do the same as Suno-Bark. What about singing though?

It is the same but Bark was the previous project of Suno before they created the music AI. With a bit of luck you can create music and also singing with it but it's quite random. Try yourself:
https://huggingface.co/spaces/suno/bark

Is it possible to save voices and always get the same voice actor back with it?

Yes you can have fixed speakers/voices however the longer the text and sentences, the more likely it deviates from the original.
And please don't credit me, the original trained models do the heavy work 😄

I would really be pissed if this was removed and I couldn't make AI Metal anymore.

You can't close Pandora's Box once it is opened.
https://www.youtube.com/watch?v=Z5EGZ145K_w