Video streamed in has green pixels/screen

[junhee071]

Hello. When I stream in video from my Ring camera using this package, if the connection from the camera to my wifi is bad, the streamed in video I see is often green (part of the screen is green/green pixels dominant the frames).
However, when I view the same footage through the Ring app, the footage is not green, but just blurred out.

I'm wondering 2 things

1. what causes the video to have green pixels when I use this package?
2. why for the same video does the Ring app have better quality (no green pixels)?

thanks!

[tsightler]

A couple of things:

1. Green pixels generally mean there is no data to decode, usually due to dropped packets as part of an I-frame (complete frame). This can be caused by poor wifi connection to the camera (packets dropped before sending to the network) or by packet loss on the receiving side due to insufficient buffers (more common on low-end hardware like RPi, and especially with Linux, which has very low default UDP buffer). For the latter issue, increasing the default buffer size can help a lot (setting net.core.rmem_max=2097152 and net.core.rmem_default=2097152 seem to work quite well for Linux).

2. Ring app uses native WebRTC support in Andriod/iOS/web browser. WebRTC support on these platform is highly developed, native code and supports advanced features such as adaptive bitrate, so it can drop to lower bitrates based on the quality of the connection. Native WebRTC is not available for NodeJS, so support is provided via werift library written entirely in Typescript. It's actually pretty amazing that you can create a usable WebRTC implementation that runs fully in a Javascript engine at all. It's a good library, but it misses some advanced features of browser/mobile WebRTC implementation which have built-in adaptive bitrate. That being said, it should be possible to implement adaptive bitrate with werift, it would just require handling manually in ring-client-api streaming code, something that just hasn't been worth the effort since Ring cameras use fairly low bitrate streams anyway (partly why their quality is pretty poor vs competitors with similar specs, I'm guessing they do this to save on recording storage costs). If you have a Ring camera with such poor wifi that it can stream it it's base rate, you really should look to improve that. That being said, we'd happily entertain PRs that implemented adaptive bitrate.

[tsightler]

It's possible 2MB isn't enough, I've seen cases where 4MB or 8MB of buffers were required, especially on low-end hardware. Also, remember these settings are not persistent across reboots unless you modify configs to set them on each boot.

It's actually quite complicated to make any other suggestions because the first step is to understand where the problem actually is and you haven't shared your hardware or even what exactly you are trying to do (are you just streaming in Homekit or doing something else directly with ring-client-api) and you also seem aware that the signal from the camera is less than ideal.

It's important to note that streams do not happen directly from the camera, they are always proxied via Ring's media servers. If the data is being dropped from the camera to those servers, there's pretty much nothing you can do on the client side because things like tcp instead of udp will only impact your connection to Ring's severs, not the connection between Ring servers and your camera. I didn't have much success trying to force TCP connections with werift back when I tested it, the code didn't even appear to support it back then, but I haven't followed the low level developments in werift.

If you are using ring-client-api directly, you can modify the streaming offer to either remove some higher resolutions, or, if your cameras supports it, HEVC/H.265 mode. The ring-client-api code does not include HEVC in the offer by default because Homekit, as well as most web browsers, will not play HEVC via WebRTC, so we only include AVC/H.264 and Ring transcodes it on the fly..

One other things I've seen that I forgot to mention, Ring servers run in AWS, and AWS has a massively multi-pathed network. I've seen a lot of cases, especially during peak network hours, where lots of packets are delivered out of order. The jitter buffer in werift seems significantly less advanced than in brower implementation. The offer does include rtx, which is a method for the client to ask for packets to be resent, which helps to compensate for this, but there's limited time for a packet to be resent since it's a low-latency real-time stream.

Overall, best best is just to make the network stable. It works! Good luck!

[junhee071]

Hi @tsightler , back again with another question.

I’ve been trying to set various FFMpeg parameters when initiating streaming through ring-client-api. These parameters are largely the resolution, frame rate, avg bite rate, max bit rate, and buffer size.

However, once the streaming has concluded and I use a tool like ffprobe to analyze the streamed/downloaded video, it seems my configuration for these ffmpeg parameters never took effect. For example, if I set the average bit rate to be 300k, the streamed video often will have an average bit rate of say 1700k.

I considered if the values for these parameters I set were too low, so the ring-client-api disregards them and injects its own values. Even when considering this and using the whole range of different values for each parameter, my specification never seems to actually take effect.

I’ve turned on ffmpeg logging by including the -report ffmpeg flag in the initiate stream call, and the resulting log shows the call made to the ffmpeg binary does have the same values as the ones I specified.

im wondering if you have any insight into this matter. Also, if it’s more convenient, I’m wondering if we could continue this conversation through email? If so, please reach me at [email protected]

Thanks for all your help so far!

[tsightler]

Again, it would really help to understand if you provided some actual code examples for whatever it is you are trying to do. You can't change the bitrate coming from the camera via ffmpeg, so it's really unclear, based on your prior messages, what you are attempting to do. You can potentially change the output bitrate, if you are using ffmpeg to transcode, but, to be honest, 300k is incredibly low for a 1080p, or even 720p, stream and the quality is likely to be terrible.

But, to be 100% clear, ffmpeg settings have absolutely zero impact on the stream you can get from the camera, only the output (and only if you transcode). If you want to try to manipulate the incoming bitrate you would need to modify the offers in the WebRTC streaming code in ring-client-api, and, even then, I've had limited success getting Ring to send other bitrates/resolutions.

[junhee071]

@tsightler i see... what i've been doing is simply streaming in the footage from my Ring camera by hooking it up to ring-client-api and invoking the RingCamera.streamVideo() method
i had been under the impression that the ffmpeg params i pass in to this method would affect the outgoing data from the sender (camera) itself, but thanks for clarifying that this is not the case

given everything i've learned, i'm left with the following problem. i'm experiencing packet loss (PL) somewhere along this route: ring camera -> ring server -> my machine streaming in the video. it's not clear exactly where on the route this PL is occurring, but it nevertheless is occurring, likely due to poor network connectivity or congestion. due to this PL, the resulting footage frequently has green pixellation.

ultimately, it will be very hard for me to improve the network connectivity between the camera <-> network. given i can't preclude the packet loss, i instead want to focus my efforts on mitigating the worsened viewing experience (seeing the green pixel visual artifacts). by this, i mean i want to remove the green color in the frame (from the green pixels). it's okay if the frame looks jagged/blurry/pixellated, as long as the bright green is gone.

to achieve this i'm thinking of 2 main directions.
the first is to write a program that detects for each frame of a video, if a large portion (say 15%) of the frame' pixels are green, then convert those green pixels to be the same color as the non-green, dominant color in the frame.

the second comes from something i've observed while playing back the streamed-in videos. the streamed video data ultimately ends up as a mp4 file. i've been playing these mp4 files through VSCode's native video player. I've noticed that for videos with this green-pixel issue, on the first playback the green pixellation is there. but on subsequent playbacks, the green pixellation is gone; the frame the green pixellation occurred in on first playback still looks a bit blurry and pixellated, but the greenness is gone. i'm wondering if i can have the functionality that removes the green pixellation enacted by default when the video is streamed-in/downloaded, so even on first playback this green pixellation is removed. would you happen to know more about how this can be achieved?

thanks for all your help!

[tsightler]

I'm afraid you more into the realm of video processing, and outside of anything plugin specific, so I'm not sure how much help I can be here, especially since you seem hesitant to share details of what parameters you are using even after asking multiple times. For example, I still have no idea what hardware you are running on, what OS, what ffmpeg options you are using, etc and all of this would impact what suggestions I might make, however, I'll give it a try with some generic options.

If you are willing to eat the costs of re-encoding (maybe you already are) there are a lot of ffmpeg options that might be useful, enabling stronger error concealment or using a deblocking filter might help, just as examples. However, if you are simply coping the streams un-modified, then options are a bit more limited, but there are still potential options, like -err_detect ignore_err -fflags +discardcorrupt or perhaps just dropping incomplete frames with -drop_pkts_on_overflow 1.

Are you sure it is really a camera side issue? Does the camera really have wifi that is this poor?