v3.9.0

[C0untFloyd]

Bugfix: Face detection bounding box corrupt values at weird angles
Rewrote mask previewing to work with every model
Switching mask engines toggles text interactivity
Clearing target files, resets face selection dropdown
Massive rewrite of swapping architecture, needed for xseg implementation
Added DFL Xseg Support for partial face occlusion
Face masking only runs when there is a face detected
Removed unnecessary toggle checkbox for text masking
Added ability to do multiple swap passes

---

This discussion was created from the release v3.9.0.

Some further infos:
Adding DFL XSeg support and the possibility of doing several passes with a fake face are in my opinion huge improvements. The XSeg masking works automatically and is very, very fast. If you're wondering what this is all about: https://www.deepfakevfx.com/tutorials/deepfacelab-2-0-xseg-tutorial/
I was thinking about having it enabled by default but chose against it. The model I'm using is a the official one from Deepface Lab, converted to ONNX. If you want to try your own, I used https://github.com/Bruc3Xu/XSegNet2onnx to convert between formats. After conversion you would place it into models "xseg.onnx". It should work right away.
Increasing the number of fake passes should be used sparingly. It sometimes affects the coloring of the face too much or gets too blurry. It's always hit and miss, some fakes looked a lot better when using 2-3 steps, most of them became too artificial looking. Experiment yourself!
Manual masking: I left it out of the regular batch processing because it works with images only. It's perfectly working but to save the result you need to save the image from the preview area.

I'm pretty satisfied with the state of roop unleashed now, the only thing missing is whole frame modification like coloring b/w images or upscaling and adding filters. With the rewritten architecture this should be pretty easy, I'm just unsure if this should be in the faceswap tab or separated into the extras tab. Also there still are many open issues and bugs with the gradio framework I'm using. Hopefully those will be fixed by them before I completely loose interest in this 😄
Anyway, I'm currently putting much work in another project fork I started some time ago:
Coiffeur
I have it running on my local machine but it's currently CUDA only and you need to be very tech savy to install it successfully. Check it out if you have an NVIDIA GPU and feel brave enough!

[lysxelapsed]

• The swapping steps option is 🔥Gives me a lot better results in a lot of cases. Nice that it works for the preview, too!
• I'll have to look into XSeg (and your other, funny project 😆) when I have some time. But as far as I understand you "draw" your own mask an then convert it to onnx? Is there a model / mask I can just edit instead of creating a new one from scratch?
• Coloring I would add to the extras tab. Filters / Upscaling sound like they deserves a place in the face swap tab in my opinion. But it gets difficult to keep the layout clean with too much stuff.
• As far as issues go, the files are still not displayed until loaded and the preview still changes at frame up / frame down only. But I think those are both gradio issues.

Amazing work as always! 👍

[lysxelapsed]

Finally. Got it to work, yay! 😃 Neither adding the cudart dll, nor pasting the pyd in the python library folder worked, same error. Since I'm not using miniconda, it additionaly needed the python311.lib in the fused folder and in coiffeur\Lib\site-packages\torch\lib. Now it works but phew it does take its time to process with a ref image - maybe less iterations would work too?

[C0untFloyd]

🎆 👍

Now it works but phew it does take its time to process with a ref image - maybe less iterations would work too?

Yeah, it's not something you can run on your average consumer pc, at least not for videos. Perhaps you noticed already, if you're using an image for the 2nd time it is much faster. The first time there is a lot of preparation going on, aligned and cropping the faces, calculating the latents etc. but for every face the different things get saved into subfolders in testimages. That is already an improvement to the main branch which cached nothing and had no GUI. But it's still just some sort of prototype to have fun with at the moment.

You once wrote you own a VR Headset, weren't you? You're not by any chance coding for it? Because currently all my coding spare time is used up for a Quest 3 XR Project using Unreal Engine 5. It's working well so far BUT like many other people I can't get UE accept my installed Android SDK for some reason. THIS IS DRIVING ME NUTS! It's working great in the Simulator but I can't package it for the real hardware. And my google foo fail me in this case, it seems lots of people have this issue but no solution so far.

[lysxelapsed]

Yeah videos would be overkill for a consumer pc. But it's ok for photos and it definitely is FUN 😄 Yes I've seen there are a lot of steps involved, but the results are really good IMO.

Yes I do, but it's an Oculus GO from the dinosaur era ^^ And no, coding drives ME nuts in general, damned typos and such make me wanna hit the chair with the little toe 🤣 I like to troubleshoot and figure stuff out - like the coiffeur stuff, was a nice exercise - but coding myself? Not so much. BUT I do have a friend who's into UE coding. I'll visit him sometime this week, maybe he knows something.

[C0untFloyd]

Thanks, meanwhile I figured it out. Android must be installed into the engine version(!!!) in addition to setting it as target in the project. Took me a almost a week and multiple Android SDK installs! This is very much still in its infancy but now I can have a virtual music visualizer in my living room 😄

it's an Oculus GO from the dinosaur era

That's stone age but not dinosaur era. I started my VR adventures with the Oculus Rift DK2. Later I also wanted to have an Oculus Go but bought the Samsung Gear VR instead because it was compatible with some hacks and I already had the necessary cellphone. The Gear VR would still give most current headsets a run for the money because it had OLED and pretty good resolution. I watched so many movies with it back then. Meanwhile all has matured and AR is of course the next big thing. I miss the old times though, there were so many relaxing and free prototypes to try out the magic of VR illusion. My favourite was flying in an air balloon with a roof above in the middle of a rain storm, a scrolling satellite google map below and a radio you could turn on. Nowadays everything looks similar Unitysomething and wants to be the next Beat Saber. But I disgress... 😃

[lysxelapsed]

Great! I was just asking myself if I would've figured that out. Probably not 😆

but now I can have a virtual music visualizer in my living room

Would be very much interested in this 😃You think that would run on the Go? I still use the Winamp visualization plug-ins sometimes - Yes, I'm robot-dialing-into-the-internet-old 😜

I miss the old times though, there were so many relaxing and free prototypes to try out the magic of VR illusion.

Indeed, there was lots of free, simple, relaxing VR stuff. Used the headset far more for that and watching movies, than games and other more "demanding" things.

[Piotrtheai]

good morning, can you help me solve this problem? I've probably tried everything. I have Visual Studio installed, antivirus programs disabled.

Launching App
Traceback (most recent call last):
File "D:\deepfake\roop-unleashed\installer_files\env\lib\urllib\request.py", line 1348, in do_open
h.request(req.get_method(), req.selector, req.data, headers,
File "D:\deepfake\roop-unleashed\installer_files\env\lib\http\client.py", line 1283, in request
self._send_request(method, url, body, headers, encode_chunked)
File "D:\deepfake\roop-unleashed\installer_files\env\lib\http\client.py", line 1329, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "D:\deepfake\roop-unleashed\installer_files\env\lib\http\client.py", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "D:\deepfake\roop-unleashed\installer_files\env\lib\http\client.py", line 1038, in _send_output
self.send(msg)
File "D:\deepfake\roop-unleashed\installer_files\env\lib\http\client.py", line 976, in send
self.connect()
File "D:\deepfake\roop-unleashed\installer_files\env\lib\http\client.py", line 1455, in connect
self.sock = self._context.wrap_socket(self.sock,
File "D:\deepfake\roop-unleashed\installer_files\env\lib\ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "D:\deepfake\roop-unleashed\installer_files\env\lib\ssl.py", line 1104, in _create
self.do_handshake()
File "D:\deepfake\roop-unleashed\installer_files\env\lib\ssl.py", line 1375, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [WinError 10054] Istniejące połączenie zostało gwałtownie zamknięte przez zdalnego hosta

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\deepfake\roop-unleashed\roop-unleashed\run.py", line 6, in
core.run()
File "D:\deepfake\roop-unleashed\roop-unleashed\roop\core.py", line 345, in run
if not pre_check():
File "D:\deepfake\roop-unleashed\roop-unleashed\roop\core.py", line 120, in pre_check
util.conditional_download(download_directory_path, ['https://huggingface.co/countfloyd/deepfake/resolve/main/inswapper_128.onnx'])
File "D:\deepfake\roop-unleashed\roop-unleashed\roop\utilities.py", line 196, in conditional_download
request = urllib.request.urlopen(url) # type: ignore[attr-defined]
File "D:\deepfake\roop-unleashed\installer_files\env\lib\urllib\request.py", line 216, in urlopen
return opener.open(url, data, timeout)
File "D:\deepfake\roop-unleashed\installer_files\env\lib\urllib\request.py", line 519, in open
response = self._open(req, data)
File "D:\deepfake\roop-unleashed\installer_files\env\lib\urllib\request.py", line 536, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File "D:\deepfake\roop-unleashed\installer_files\env\lib\urllib\request.py", line 496, in _call_chain
result = func(*args)
File "D:\deepfake\roop-unleashed\installer_files\env\lib\urllib\request.py", line 1391, in https_open
return self.do_open(http.client.HTTPSConnection, req,
File "D:\deepfake\roop-unleashed\installer_files\env\lib\urllib\request.py", line 1351, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [WinError 10054] Istniejące połączenie zostało gwałtownie zamknięte przez zdalnego hosta>

Done!
Press any key to continue . . .

[C0untFloyd]

#486

[C0untFloyd]

It's the very same reason and please don't pollute this thread with non-issues but search through the existing ones. There's a very high possibility somebody else already had your problem, see above.

[RoopUser]

A guide on masking within the current interface would be great... it seems like we need to do keyframing of masks, but I am not sure how layers come into play (and how a single mask at some point affects a certain part of the timeline or the WHOLE timeline), etc.

[Walfer9]

Was wondering how the new masking works. I am still relatively confused about how Xseg works on how it masks on the program and I was not able to get it to work properly.

[lysxelapsed]

I think a lot of people overlooked that the manual masking only works for images, not videos.
As for Xseg: Didn't do a whole video with it. But my guess is, from the time it took to generate preview frames, that it will be a lot slower than without masking. Did a lot of comparing with preview frames and it looks good in, I'd say 50% of the frames I compared. For the other half however, masking everything out of the face area looks either unnatural / weird, or you don't see much of a difference at all. But you can easily compare for yourself by saving the preview frames.

[Walfer9]

TBH I am not entirely sure how xseg works since I am relatively new to the whole thing. Cuz I saw what Xseg is doing and basically highlighting the portion of the face in which it could be replaced but I do not know how to do that part.

[lysxelapsed]

It's an already pretrained model / mask. You just have to select it, thats all. If you want to create your own, follow the link in the first post.

[Walfer9]

Maybe I'm doing something wrong. I put Xseg masking on and saw the preview of the masking. However, when I try to switch, the face still clips as if the Xseg masking was turned off.

[Pr0f3ssi0n4ln00b]

Wait, how does Xseg work? Noob here. Do I just change to Xseg masking and that's it? Or does the setting not matter unless choosing manual masking?
Or does it mean I have to do manual masking and moving vectors etc for every face in my faceset.fsz? And then do training based on that again? I'm a little lost in the process.

Is it possible (or an idea at all?) to add facesets for target faces as well? Would that increase accuracy on the target face?

Running CPU only fwiw.

Oh and thank you so much for this repo, it works way better than I expected when I found it! Considering its relative ease of use.

[C0untFloyd]

Do I just change to Xseg masking and that's it?

Yes. It's a trained Net to detect and mask face occlusion automatically.

Is it possible (or an idea at all?) to add facesets for target faces as well? Would that increase accuracy on the target face?

No and no. However it's planned to save the face selections into some collection for latter use, so that you don't need to redo the face selection if you process the same video/images again.

Oh and thank you so much for this repo, it works way better than I expected when I found it! Considering its relative ease of use.

I'm very glad if that's your opinion because lately there are so many people opening issues about problems, where the solution mostly is right in front of their eyes and the real problem is sitting in front of a monitor 😄

[Pr0f3ssi0n4ln00b]

Does the occlusion masking work primarily on source or target?

I.E: Perfectly visible source face to occluded target vs occluded source face to perfectly visible target.

[C0untFloyd]

Target only. The order is Source Face -> Target Face -> Mask occluded areas on target face -> Re-apply occluded target face areas to final image. That way a hand in front of the face won't be pasted over with parts of your fake face.

[Walfer9]

The last part is the one I am confused. I chose Xseg as the masking option and looked at the preview. I see where the outline is but every time I run the program it does not mask out properly as if Xseg did not work. I am still relatively new to the program so not entirely sure how it works when applying the end product.

[thegreek1990]

my directml has tanked since the update and will only recognize CPU. "AMD always giving a headache >.<"

before only when using masking it would only use CPU instead of Directml. Has anything been automatically implemented? I can't even get it to show up now. I tried to force some dml/directml in the settings but it automatically resorts to Cuda now.

[C0untFloyd]

If you're on windows and using the automatic installer to update it probably overwrote the onnxruntime with the cuda one. You would need to re-install the directml one.

[thegreek1990]

You hit the nail on the head.

I went into

C:\Users\username\AppData\Local\Programs\Python

I saw another python in there, not related to the one I have installed/path etc I deleted that one completely.

I then went into

C:\Users\username\AppData\Local\Programs\Python\Python310\Lib

deleted any onnx inside. ran pip uninstall onnxruntime onnxruntime-directml just incase.

then ran pip install onnxruntime-directml to update to the latest directml. (1.17.3 latest)

works perfect now.

thankyou C0untFloyd

Also Xseg works with DML amd graphics. 😄

[maltaannon]

I'm getting the most unexpected error. Installing requirements.txt went without a hitch. For some reason I get:

Traceback (most recent call last):
  File "E:\roop-unleashed\run.py", line 3, in <module>
    from roop import core
  File "E:\roop-unleashed\roop\core.py", line 22, in <module>
    import roop.utilities as util
  File "E:\roop-unleashed\roop\utilities.py", line 11, in <module>
    import gradio
ModuleNotFoundError: No module named 'gradio'

As if gradio is not installed. However, when I do:

import gradio as gr
print(gr.__version__)

I'm getting 4.38.1 so everything seems to be ok. This is all in a venv. Any ideas? Thanks.