Support HLS Fallback for WebRTC cameras

[OnFreund]

Cameras that support WebRTC provide lower latency and smooth playback. They do have on major downside though - they're not supported as media source.

I propose we allow WebRTC cameras to fallback to HLS, so that WebRTC is used when the camera is streamed by the HA frontend, and HLS when it's consumed as a media source.

The code change is pretty minimal - adding another value to StreamType to represent the new type, and changing this if statement and this one accordingly, and a couple of if statements on the frontend. (for completeness, this if statement can also be changed to allow better error handling).

Cameras that want to benefit from this change will use the new StreamType, and implement both stream_source and async_handle_web_rtc_offer, rather than just one of them.

[allenporter]

See the initial discussion for this feature it was decided not to allow supporting multiple stream types, but instead force cameras to support one or the other: #640 (comment)

-- but does it already work this way after home-assistant/core#80646 ?

[OnFreund]

Thanks @allenporter.
That discussion took place several months before the introduction of camera media sources, which I believe changes the picture drastically, justifying another discussion.

As for the PR, it's what allows this fallback to work (and another point in favor of revisiting the discussion, since it also supersedes the discussion), but the media source browser excludes WebRTC cameras. That's why I suggested changing this if statement.

[OnFreund]

That PR perhaps made some services work(I don't know), but webrtc cameras are still missing from the media sources browser.

[allenporter]

OK. I guess my take is: We've already decided that supporting HLS is OK for the media source, and so it seems reasonable to me to have these media sources show up in the media browser. I don't think we need to (nor should we) make the proposed changes to the entity model to change the StreamType, however so not sure i agree on the proposed implementation details.

On use cases: What integrations are the target to take advantage of this? I know that nest is moving some more cameras to webrtc, though i'm not sure rtsptowebrtc is seeing massive of adoption necessarily, and i don't think anything else supports it.

[OnFreund]

Thanks. I'd love to see a suggestion that doesn't include changing StreamType, but I do think it should be explicit, rather than showing a camera in the browser only for it to fail.

The Nest integration was enough to add WebRTC support to HA, so I think it alone can justify this change. My personal main motivation is a non-core integration - Frigate, but the forums have a lot of discussions with people wanting more WebRTC support.

[allenporter]

OK, my impression was that frigate doesn't use any of this...

[OnFreund]

The frigate integration recently added WebRTC support, and can potentially fallback to HLS (which is what it uses anyway when WebRTC is disabled in the integration options).

[OnFreund]

Seems like (at least some) WebRTC Nest cameras can supply HLS streams.

I also created a PR to allow Frigate cameras to stream HLS even when they're set up as WebRTC cameras.

Maybe we can just show WebRTC cameras in the media browser, and worst case some Nest cameras will fail?

[allenporter]

Is there a fundamental reason why media browser could not be updated to support webrtc cameras? Is this a limitation in core or just not code in frontend to do it?

[OnFreund]

Do you mean real support over WebRTC? I don't know enough about WebRTC to answer.

If you mean updated to support them by falling back to HLS then that's what I'm proposing - it's just a single if statement in core that needs to be changed (I have it working locally).

[OnFreund]

(just to clarify: for real support it's not just the HA frontend - this would also need to be implemented for cast and other media players)

[OnFreund]

Since this change is relatively small, and has no bearing on the entity model, I went ahead and created a PR for it. We can keep this discussion open to try to find the right long term approach for WebRTC and media sources.

[OnFreund]

Seems like the new WebRTC implementation is breaking the HLS fallback for WebRTC cameras.

What's preventing us from having the fallback work again?
cc @edenhaus

[allenporter]

The APIs are documented under WebRTC streams and WebRTC Providers https://developers.home-assistant.io/docs/core/entity/camera/#webrtc-streams. Fallback works when you start with an HLS stream and use a WebRTC provider to make it support WebRTC. I don't see that multiple layers of conversion would be involved. If a camera is already webrtc then the provider won't be used. My impression is that all the necessary APIs are provided.

[OnFreund]

I don't see that multiple layers of conversion would be involved. If a camera is already webrtc then the provider won't be used

If the camera supports both WebRTC and and RTSP and HA is forcing it to choose one, then multiple layers will potentially be involved, as in this case. Additionally, there are points 2 & 3 above.

I went over the docs and found:

By implementing the WebRTC methods, the frontend assumes that the camera supports only WebRTC and therefore will not fallbac to HLS.

What I'm trying to figure out is the impetus for this limitation. We know it's technically possible (this is what I added in #108796), and I was under the impression that the new webRTC implementation deals with RTSP cameras, and shouldn't affect existing WebRTC ones.

[edenhaus]

Some clarifications:

Seems like the new WebRTC implementation is home-assistant/core#129990 (comment).

The linked issue is that the custom component frigate is not using the camera entity correctly. There was no fallback to HLS for native WebRTC cameras before 2024.11 and not after it. So, the WebRTC implementation is not breaking anything.

Inefficient - HA adds another WebRTC layer via go2rtc, despite the camera already supporting WebRTC.

That's incorrect, as Allen has already mentioned. The go2rtc provider will only be used if the camera doesn't support natively WebRTC.

Forces the user to have two separate firewall/router configurations, for consuming the camera directly, and for consuming it through HA.

Go2rtc was added newly to Home Assistant, and we can't take into account custom components as they are out of our control and we can't guarantee support for them. So this assumption is wrong as not everyone uses frigate. We will not stop adding a great feature for all just because some custom component is using the same binary. Also nothing is preventing frigate to be added as core integration.

Potentially sets up a port conflict, and overall increases the friction.

The core would always win on port conflicts. We cannot control which ports addons are using. To avoid issues for frigate addon users, core prefixed all ports with a 1 as discussed with the Frigate maintainer. Nevertheless, port conflicts can always happen, and core has nothing to do with it. For example, by installing the go2rtc and frigate addon, you also have a port conflict, and in this case, frigate will lose as the go2rtc addon is started first.

Your points 2 and 3 refer to everyone using frigate, which is incorrect. We don't know, as Home Assistants, what frigate is using as a custom project. And due to the massive amount of custom components, it's not feasible to check any of them.
The frigate maintainer can always choose to move their custom integration to the core by contributing it to the core.

By implementing the WebRTC methods, the frontend assumes that the camera supports only WebRTC and therefore will not fallbac to HLS.

What I'm trying to figure out is the impetus for this limitation. We know it's technically possible (this is what I added in #108796), and I was under the impression that the new webRTC implementation deals with RTSP cameras, and shouldn't affect existing WebRTC ones.

The frontend_stream_type could technically only return a single enum value; therefore, it was impossible to specify two protocols. This is solved in HA 2024.11 by adding the camera capabilities, which can return a set. As the name already implies, this property is for the frontend. As you are quoting me outside the context, please let me add the needed context. The above was written in term of frontend and we currently don't support a fallback to HLS on native WebRTC cameras as a camera entity doesn't need to provide a stream source.

As this architecture discussion is still in progress (not answered yet), I did not take it into account and as open source project we are happy for any contribution

[OnFreund]

Thanks for the detailed response.

There was no fallback to HLS for native WebRTC cameras before 2024.11 and not after it. So, the WebRTC implementation is not breaking anything.

I'm not sure what you mean by "native", but Frigate cameras did fallback to HLS before 2024.11 (that's the topic of this discussion). Your following points rely on this assumption that cameras don't fallback.

The frontend_stream_type could technically only return a single enum value; therefore, it was impossible to specify two protocols.

Prior to 2024.11, we simply checked whether the camera also supported a stream to allow the fallback. Not ideal, since the call could block or fail, but it worked.

The above was written in term of frontend and we currently don't support a fallback to HLS on native WebRTC cameras as a camera entity doesn't need to provide a stream source.

Once again, this relies on no fallback - but Frigate is a case where I believe it's needed.

As this architecture discussion is still in progress (not answered yet), I did not take it into account and as open source project we are happy for any contribution

I'm happy to do that, but I still don't understand the reason for the objection - what's the fundamental issue with supporting the fallback? So far all the response are "it's not needed", but Frigate is a case where it is.

[allenporter]

Architecture discussions are usually not targeted to support custom component exclusive features directly. Usually things are designed so that custom components can use the same APIs and get similar behavior, which is what is happening here, but extending core for things core does not use is not common.

Are we really talking about an actual device with native webrtc and rtsp support? If this is just talking about go2rtc still, but run in a custom component, it seems like it's a matter of using the provider APIs.

[OnFreund]

Architecture discussions are usually not targeted to support custom component exclusive features directly.

I agree with this, but I think this case is a bit different because:

1. The principle is the same for any NVR that can serve WebRTC, Frigate is just an example.
2. It's an extremely popular custom integration, probably more than most of the core camera integrations. In fact, back when the HA discord used to have a cameras channel, most of the conversation there was Frigate related.
   I believe the only reason Frigate is not a core integration is legacy, and now it's too big to port. We'll have to port it in small increments, and every time we'd do that, the custom integration will have changed by the time it's merged, creating an Achilles and the tortoise type race. I think the best case estimate for convergence is 6 months.

Are we really talking about an actual device with native webrtc and rtsp support?

I believe Frigate passes as an "actual device", but I don't see why it matters - from HA's perspective, there are cameras that support WebRTC and RTSP. Frigate using go2rtc is an implementation detail that is not exposed to HA, and can change. Note that it's not the integrations, but rather the NVR itself. Imagine this was a hardware NVR. The integration does use the relevant APIs, and supported the both WebRTC and HLS up until 2024.11.

[allenporter]

@edenhaus:
I was able to find 3 core integrations that implement WebRTC. I don't know whether Ring or Nest support RTSP as well,

These cameras don't need any fallback. In nest webrtc works fine without anything else (plus they can be native with now rtsp supported)

@allenporter:
Reading this thread again, I'm a little confused as to why you suggest using the provider API. From what I understand, it's used by integrations that want to provide a WebRTC wrapper over any camera stream, not just the integration's cameras. That seems to be a very roundabout way to handle this, and will mean that the Frigate integration will have to support arbitrary RTSP streams. What am I missing?

I thought you initially said your requirements are for an integration using go2rtc which is a webrtc wrapper over a camera stream. Anyway, I am not planning on discussing custom components here anymore.

[OnFreund]

plus they can be native with now rtsp supported

Can you clarify this?

thought you initially said your requirements are for an integration using go2rtc which is a webrtc wrapper over a camera stream.

I think there might be some misunderstanding regarding the requirement here, which is for cameras that natively support both webrtc and rtsp to serve their native webrtc when possible, and fallback to their native rtsp stream when needed (e.g. when casting). The Frigate integration does not use go2rtc, and the fact that Frigate uses go2rtc is an implementation detail that's not exposed.
This requirement was satisfied before 2024.11, and apparently is still satisfied today, but for a new line in the documentation that says it isn't. I haven't been able to get a reason for this documentation change.

[edenhaus]

This requirement was satisfied before 2024.11, and apparently is still satisfied today, but for a new line in the documentation that says it isn't. I haven't been able to get a reason for this documentation change.

First, you are referring to my comment on the issue and not documentation.
Until 2024.11, the frontend only used the player defined with the frontend_stream_type. There was NO FALLBACK implemented for any camera, as the frontend has no support for it. So there is no way that Frigate had a fallback before 2024.11 (except you are using some custom card, which of course is out of the scope of your argumentation)
With the go2rtc provider, we added WebRTC for HLS cameras, where the frontend fallbacks to HLS if there are problems with WebRTC. We did not remove any fallback for native WebRTC, we only did not add it.

I was able to find 3 core integrations that implement WebRTC. I don't know whether Ring or Nest

Fall backing to HLS is only required if WebRTC has some connection issues. For Nest and Ring we don't need a fallback as the manufacturers make sure that WebRTC is always working (I assume their WebRTC camera endpoint is always reachable, or they provide a TURN server themself)

From go2rtc:

As you can see in the screenshot above, HLS is the worst for live streaming. We also need to find a good reason to implement a fallback to HLS for native WebRTC cameras. Please also note that in 90% of the cases, WebRTC is able to make a connection with a STUN server (provided for free for all HA users), and Nabucasa cloud subscribers have a TURN server available, so a connection can always be established. I'm not familiar with frigate and how the use WebRTC, but if you have a lot of connections issues on WebRTC there, that you probably should report it there. The frigate component could also use the ICE servers available in HA but I'm happy go give support if they have migrated their component to core

[OnFreund]

Ok, now I'm certain we have some misunderstanding.

First, some facts:

1. Fallback definitely worked in versions prior to 2024.11. I myself contributed a PR to facilitate that.
2. There's no custom card involved in said fallback.
3. I am referring to the documentation, that says:

the frontend assumes that the camera supports only WebRTC and therefore will not fallbac [sic] to HLS.

The following statement makes me believe we're talking about two different things:

Fall backing to HLS is only required if WebRTC has some connection issues. For Nest and Ring we don't need a fallback as the manufacturers make sure that WebRTC is always working

That's an unrelated concern. Fallback, as described in the beginning of this discussion, refers to the ability to use HLS in contexts where WebRTC isn't supported (e.g. when casting). The only difference between Frigate and Nest/Ring in this sense, is that Frigate can also serve an RTSP stream that can be used over HLS, instead of relying on HA mechanisms to provide that translation. All we ask is that HA use the native WebRTC implementation for Frigate cameras when consumed through the HA UI, and the native RTSP stream provided by Frigate when casting. This has definitely worked before 2024.11, and seems to still be working.

I think there was a 3-way misunderstanding here, and we're more aligned than it seems. The Frigate folks raised two concerns, and the answer to one was interpreted as the answer for the other. The only misalignment is that quote from the documentation.

[OnFreund]

To further clarify, the Frigate integration supplies two types of cameras (there are more, but I'm simplifying for this discussion):

1. Cameras that support both WebRTC and RTSP
2. Cameras that only support RTSP

The objective is not to have a fallback between 1 & 2, but rather to make sure that 1 can work in a way that:

1. Allows the native WebRTC implementation to be used in the UI
2. Allows the native RTSP to be used when casting, or any other context that doesn't support WebRTC.