0.3 update thread (WIP)

[SirLynix]

Some update about the next obs-kinect version!

General stuff

I massively refactored the code, and improved performance. You can now have multiples Kinect v2 sources at the cost of one (Kinect images/color to depth mapping will only be done once instead of once per source, dramatically improving performance). I'm not sure if that's gonna be helpful but still, it's an improvement.

I also added body and depth hybrids (as requested by someone on reddit):

• "Body or Depth" (everything in the depth range or a body)
• "Body within Depth" (a body inside the depth range)

One of the big stuff about the refactor is also the support for multiple Kinects from multiple frameworks!

This is done by having all code related to a Kinect runtime (or even all Windows specific stuff) in a separate .dll which is loaded by obs-kinect at startup. This was made to add support for freenect(2) in the future, making the plugin available on Linux and macOS (won't be in the 0.3).

There's also a new "Kinect service priority" combo box, which tells obs-kinect to raise the priority of the KinectService.exe process, this is helpful in case where your CPU is under heavy load (like compiling/playing games) and prevents/reduce lost images. (this is only an issue with Kinect runtime 2.0, afaik all other frameworks, including Kinect runtime v1.0 and freenect, don't have a separate service for this).

Also:

• I fixed depth/infrared distortion, changing to depth/infrared source now resize the Kinect source to the right size
• I added a "dirty depth" max count setting, which allows the source to fetch a depth value from a previous frame if it doesn't have a value for some pixel. This helps with flickering but introduces a "depth lag" in case of movement.

About Kinect v1

As you can see, I'm now able to retrieve Kinect v1 color and depth streams. I'm even able to retrieve both of them at the same time (which isn't as straightforward than with Kinect v2) and try to do some green-screen stuff.

... Okay that part still needs some work.

Interesting notes about Kinect v1:

• It's a pain in the ass to use compared to Kinect v2 SDK.
• It has lower color/depth quality than v2 (duh)
• It has an accelerometer and a motor allowing to look up/down on its own.
• It has a "near mode", which brings the depth range from 80-400cm to something like 40-200cm
• It works by exclusive access, which means you can't have obs-kinect and another process using the same kinect at the same time (you can still have multiple obs-kinect v1 sources).
• You can have multiple Kinect v1 at the same time, obs-kinect will ask you which one you want to use (warning: afaik only the first one can have body stream due to a SDK limitation).
• It allows to change white balance/contrast/hue/saturation/exposure/brightness/gain/backlight compensation settings. Kinect v2 doesn't allow that unfortunately (but I'm looking to add support for it through raw USB commands, since the device does support it).
• It has a dedicated background removal stream

That last one was a surprise for me, it also works quite well (even better than my own background removal with Kinect v2, except for the color quality).

Here's an example from Microsoft SDK:

I wish to be able to have something this smooth for Kinect v2 one day.

Anyway, I'll expose this new source for Kinect v1 (unfortunately v2 doesn't have that).

I have no idea when 0.3 will be out, things complicated a lot because of all the Kinect v1 possibilities (and because it's working in such a different way from Kinect v2). I'll update this thread if I have anything relevant to say about this.

I'm sometimes streaming obs-kinect dev on Twitch

I worked on obs-kinect 0.3 body/depth hybrids in english some days ago, you can see the replay here:
https://www.twitch.tv/videos/605065358

I also worked on the Kinect v1 support, in my native language (french) here:
https://www.twitch.tv/videos/607021382

Enjoy!

[SirLynix]

Body and depth filtering work.

https://twitter.com/SirLynix/status/1256359447947083779

Depth filtering is a lot more unreliable than with Kinect v2 though.

[JonathanFly]

A lot of progress! I'm a bit disappointed in V1 RGB quality but I knew the resolution was low. The background removal is impressive though.

It allows to change white balance/contrast/hue/saturation/exposure/brightness/gain/backlight compensation settings. Kinect v2 doesn't allow that unfortunately (but I'm looking to add support for it through raw USB commands, since the device does support it).

Oh, that would drastically improve general webcam usability, hope you can get that to work.

I'm too lazy to build this myself but if you post a beta build I can try both Kinects this week.

[SirLynix]

A lot of progress! I'm a bit disappointed in V1 RGB quality but I knew the resolution was low. The background removal is impressive though.

I though it was going to be worse than that but yeah, it's outputting a 480p color image (opposed to the 1080p of the v2).

Kinect v1 does support 1280x960 color images at 15 FPS, I'll add support for this as well.

Oh, that would drastically improve general webcam usability, hope you can get that to work.

I think I'll be able to do that on v2 by using a library made by Microsoft (found here), unfortunately it's not open-source and is only compiled for x64, which means I won't be able to do that on the 32bits version of this plugin (shouldn't be a big deal though, I think almost everyone uses a 64bits OS now).

I'm too lazy to build this myself but if you post a beta build I can try both Kinects this week.

I will! 0.3 is really a big update and will require a lot of testing before being out. You can expect a release candidate release in the next few days!

[SirLynix]

Example of 1280x960 output from Kinect:

[SirLynix]

I added support for infrared output.
It seems Kinect doesn't like having both infrared and body frames at the same time though (edit: it doesn't like color and ir at the same time neither).

[SirLynix]

I feel stupid.

I managed to halve frametime by... not processing the same image twice.
Yup. I forgot to do that.

Fixed in aac42ca

[SirLynix]

Release candidate binaries are coming soon!

[SirLynix]

Dedicated background removal (using Kinect Developer Toolkit background removal) is now supported for Kinect v1.

This is the best background removal I can provide for Kinect v1, Kinect v2 won't have it (due to Microsoft not releasing a similar tool for it).

I'm now working on the translations keys and then proceeding to release obs-kinect 0.3 RC!

[SirLynix]

First release candidate for v0.3 is out!
Please test it and report any bug you encounter by creating an issue.

[SolidSouless]

I just tried this addon with Kinect V1 360 and It perfectly maps me out of the picture and leaves the background, I can't seem to reverse it so the background is removed.

[SirLynix]

Haha, first Time I hear that.
Could you please post a screenshot so I get to see what it looks like? Could you also post all the settings you're using?

[SolidSouless]

Well, I just found out it was because I was too close! When I back up a bit more, It works fine. lol Here are the screenshots of what it looked like and the settings I had.

[SirLynix]

Oh right, the Kinect for 360 cannot read depth below 80cm in front of it (up to 400cm), Kinect for Windows can bring this range to 40-200cm by enabling "near mode". Kinect v2 (xbox one) has a range of about 50-500cm.

There's nothing I can do against this, as it's an hardware limitation.

[SolidSouless]

Nah I get it! It was just my own stupidity, It works really well! Props to u my dude 👍

[SirLynix]

Hello guys. It's been a few days since 0.3 RC2 got out and I fixed some minor stuff already for RC3/0.3 release.

Did you encounter some bugs/crashes I could fix before releasing 0.3? Please tell me, I don't want to have to release a 0.3.1. 😄

Also, there's one or two keys missing in the german translation, could someone take a look at that?

[JonathanFly]

I only tried V2 for a few minutes. No bugs or crashes. The main thing I noticed is my green screen error is actually pretty consistent, in my case shifted to the left. If I could nudge the cutout by X percent or pixel that looks like it would dial it in pretty well.

[SirLynix]

The main thing I noticed is my green screen error is actually pretty consistent, in my case shifted to the left. If I could nudge the cutout by X percent or pixel that looks like it would dial it in pretty well.

That's weird. Could you show me what it looks like?

[delet-1]

I'm using a V1 Xbox 360 version and camera options like auto-exposure, brightness, exposure time, frame interval and gain doesn't change anything. Camera elevation does work correctly.

[SirLynix]

Did you try to disable auto-exposure and play with the other options?

I fixed a bug recently which prevented device-specific settings to be applied at device acquisition (when the source shows up, when OBS is started, etc.).

[delet-1]

Yes, I've tried it with different combinations.

[SirLynix]

Okay so you have a Kinect for 360, maybe that's the same issue as #13 and changing camera parameters is not supported on that model. Could you try to use Microsoft Developer Toolkit Browser and run the Kinect Explorer example? It has some options to controls those settings.

If that doesn't work I guess I can't do much. I wonder if libfreenect is able to do this.

Edit: Oh I forgot to ask, do you have errors in OBS log?

[delet-1]

No errors in the logs. I've checked Kinect Explorer and the only options are related to resolution or picture type (rgb, yuv, infrared, rawbayer), no mention of brightness or such.

[SirLynix]

Here, try adjusting exposure settings to see if it's changing something.

[delet-1]

The option is missing so I guess Xbox 360 v1 doesn't support it at all.

[Milliw]

RTX 2060

[SirLynix]

Maybe try updating your drivers, I'm having a RTX 2080 Ti and no issue. The logs seem to be pointing a driver crash. It would be easier if I could reproduce any of theses crash myself.

[Milliw]

I´m on 460.79. Should be quite recent. Wanted to update/look up for the newest driver but I can´t complete the form on the nvidia page. Tried three different browsers... guess nvidia doesn´t like me

[Milliw]

Updated to last 460.89 WHQL by downloading it from third party site. Still same issue. Works when I first open the settings and crashes then when changing the background mode several times. Not a big issue maybe. I can work around.

Great update btw! ;)

[SirLynix]

Thanks! All I need to do now before releasing this as the 1.0 version is to make it work with Linux using libfreenect. Shouldn't be that complex 😄

[Milliw]

One more note:
The cameras I have in OBS from earlier version of your plugin don´t work out of the box. Only after removing/rechecking the greenscreen imitation checkmark it does.

Removed all old Kinect sources. Added a new one. Changed background effect -> crash.
Thought removing old ones could maybe solve the issue.

[Milliw]

Unticking the checkbox to use gpu to compute color-to-depth-calculation leads to some weird background removal. The cut area stays the same as before and not adapting to movement anymore. (Wanted to try to let the cpu do the workload as my gpu is not the strongest)

(Kinectservice.exe uses up to 14% and obs up to 12% gpu load according to task manager. Without streaming. Quite heavy load in my oppinion.)

[SirLynix]

I'll take a look at that, thanks for letting me know

[fotiDim]

@SirLynix congrats on the great work! After testing the build you posted above I spotted the folllowing issues:

• Crashes (sometimes) when switching the background mode (etc. blur). I am attaching some crash logs
• I am not sure how manual white balance mode works. I would like to be able to set the light temperature in Kelvin.
• I am not sure what the "Unknown" white balalance mode does.
• Most drop downs have values with key names (e.g. ObsKinectV2.WhiteBalanceMode_Auto)
• Switching between 50hz and 60 hz mode does not seem to have any effect. I do see some flickering on the camera feed.

Crash 2020-12-27 23-44-34.txt
Crash 2020-12-27 23-34-55.txt
Crash 2020-12-27 23-35-29.txt

[SirLynix]

Hi, thank you for reporting this!

Crashes (sometimes) when switching the background mode (etc. blur). I am attaching some crash logs

It still looks like a driver crash, this is very strange. I tried to reproduce this issue without success, I'll try with another computer soon.

I am not sure how manual white balance mode works. I would like to be able to set the light temperature in Kelvin.

I would love to be able to expose this, as the other Kinect models work this way too, but the Kinectv2 expects red, green and blue gains. If I had a way to convert from Kelvin to RGB I'd do it (maybe you know a way?).

I am not sure what the "Unknown" white balalance mode does.

That's why it's called "unknown" 😄. It's a valid white balance mode (as it's accepted by the Kinect) but doesn't seem to do anything different from manual without red/green/blue gains conf.

Most drop downs have values with key names (e.g. ObsKinectV2.WhiteBalanceMode_Auto)

Did you try the latest version? (obs-kinect-locales.zip)

Switching between 50hz and 60 hz mode does not seem to have any effect. I do see some flickering on the camera feed.

I'm not sure I can do anything about that to be honest, but I'll try!

[fotiDim]

@SirLynix it turned out I was not on the latest version. Everything seems to work now. Even the crashes are gone.

I was wrong about 50/60hz mode. I tested with another light and it does make a difference when changing the setting.

Regarding conversion of color temperature to RGB gains it seems this is what is needed. I tried putting it in a spreadsheet but my calculation seems to be wrong somewhere.

[pinecones-sx]

huge difference with the newest update. it's much better at not cutting out stuff it isn't supposed to.

for whatever reason it didn't work on 50hz mode (kinect v2) and i'm wondering if you wouldn't want to default this to 60hz.

[SirLynix]

So you mean you had to change the powerline frequency to have an output? What was happening before that?

Did something shows up in obs logs ?

[pinecones-sx]

actually I think I got regular video output using 50hz, but trying to use the green screen effect produced no output until I switched it to 60hz. didn't check the logs, not super familiar with that but if you want me to check something I can!

…

On Tue, Jan 19, 2021, 2:00 AM Jérôme Leclercq ***@***.***> wrote: So you mean you had to change the powerline frequency to have an output? What was happening before that? Did something shows up in obs logs ? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#7 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADKOQBAJMREHKL7LA5YQ4STS2U33NANCNFSM4MXBEOIA> .

[fotiDim]

How can I build the latest version? Are there instructions somewhere?

EDIT: never mind just saw it in the readme

[SirLynix]

Yes, the readme has a "How to build" section. But it requires a lot of dependencies (at least one per kinect model + obs-studio source code).
You can find the latest test build version here: obs-kinect-locales.

[Weirdo1312]

@SirLynix I'm not sure if i make any sense at all
but can you somehow enable "near mode" for green screen in regular xbox 360 kinect by completely ignoring the minimum distance instead of setting it to lowest allowed value, and only apply the maximum distance value?

[SirLynix]

I'm not sure to understand. Near mode is already enabled by default for xbox 360 (I think I still have to make a checkbox for that). Which brings the minimal readable depth to ~40cm in front of the Kinect, which means the minimum distance won't have any effect as long as its value is below 400mm.

[Weirdo1312]

I'm not sure to understand. Near mode is already enabled by default for xbox 360 (I think I still have to make a checkbox for that). Which brings the minimal readable depth to ~40cm in front of the Kinect, which means the minimum distance won't have any effect as long as its value is below 400mm.

isn't this type of near mode only usable on kinect for windows v1 and kinect v2?

[SirLynix]

Near mode is something that only works with Kinect v1 (360 and kinect for Windows v1), I'm not aware of Kinect v2 having that feature.
Regardless, the minimum distance of the green screen filter and is not directly linked to the minimum depth distance of the device.

[SirLynix]

I've managed to setup nightlies builds for Windows and Linux.

Nightlies builds are the most up-to-date builds of obs-kinect, with every feature I'm working on (and every bug I've yet to fix too!).

Select the most up-to-date (at the top) build that succeeded and download the "releasedbg" artifacts.

Windows
Linux

About 1.0 (and stable version): I've got to work on the libfreenect backend (for Kinect v1 on Linux) and finish the libfreenect2 backend (it only needs minor stuff like camera controls and such).

[Weirdo1312]

I've managed to setup nightlies builds for Windows and Linux.

Nightlies builds are the most up-to-date builds of obs-kinect, with every feature I'm working on (and every bug I've yet to fix too!).

Select the most up-to-date (at the top) build that succeeded and download the "releasedbg" artifacts.

Windows
Linux

About 1.0 (and stable version): I've got to work on the libfreenect backend (for Kinect v1 on Linux) and finish the libfreenect2 backend (it only needs minor stuff like camera controls and such).

Hi, I wanna package it for arch linux, but I'm not sure exactly where should i place the files
there are two obs-plugins folder and one of them is in data folder, should i put both in them obs-plugins or what?
i'm confused

[SirLynix]

Hi.

In the archive you'll find a data/obs-plugins folder, I suppose it's one of the folder you're talking about. The other obs-plugins folder contains some of the plugins binaries, the other binaries should lie next to the obs-studio executable.

[Weirdo1312]

Hi.

In the archive you'll find a data/obs-plugins folder, I suppose it's one of the folder you're talking about. The other obs-plugins folder contains some of the plugins binaries, the other binaries should lie next to the obs-studio executable.

so
"/bin/64bit/" files in archive should go to "/usr/bin/"

"/obs-plugins/64bit/" files should also go to "/usr/bin/"

"/data/obs-plugins/obs-kinect/" files should go to "/usr/share/obs/obs-plugins/obs-kinect/"

correct?

[SirLynix]

Except for plugin binary files i'd say yes.

There's probably a obs-plugins folder somewhere, can't say where to find it (maybe try looking where other OBS plugins install?)

[Weirdo1312]

Except for plugin binary files i'd say yes.

There's probably a obs-plugins folder somewhere, can't say where to find it (maybe try looking where other OBS plugins install?)

i placed them like this and it seems like it doesn't detect any device (i have kinect v1)
is it the bug of the the software or i did something wrong?

[Weirdo1312]

other plugin binary files were here that's why i thought it's a good idea to put them there but it doesn't seem to be working

[SirLynix]

Yes it seems it's at the right place.

However the reason it's not working is because obs-kinect doesn't support Kinect v1 on Linux at the moment, it's on my todo list and should be quick to add but I have a ton of works on other projects. Will try to do it soon!