Low resolution on some Android phones #133

holzgeist · 2024-07-15T06:36:52Z

Hi there,

during development of our app that uses Livekit as the video call framework, we discovered that the video quality is very low on some Android devices. I tracked down at least one bug (and a combination of heuristics combined with some bad luck I guess) in WebRTC, so I'm reporting this here and not in https://github.com/livekit/client-sdk-flutter/, the entry point we use.

The issue

We set the defaultCameraCaptureOptions to VideoParametersPresets.h720_43 (i.e. 960x720) with simulcast enabled. On some Android phones, the sent simulcast layers are 144x192 and 288x384, even on local network with a local livekit server. This is ~16% of the requested resolution, and definitely not suitable for high quality video conferencing.

The deep dive into WebRTC

After ruling out network issues, I started a deep dive into client-sdk-flutter and subsequently into WebRTC itself. Here are my findings (on a Fairphone 5 with Android 13):

Livekit room options:

RoomOptions(
  adaptiveStream: true,
  dynacast: true,
  defaultAudioCaptureOptions:
      AudioCaptureOptions(deviceId: _settings.audioDeviceId),
  defaultCameraCaptureOptions: CameraCaptureOptions(
      deviceId: _settings.videoDeviceId,
      params: VideoParametersPresets.h720_43
)));

Camera2Session lists following available capture formats: [4080x3060, 4000x3000, 4000x2250, 3960x1760, 3840x2160, 3264x2448, 3072x3072, 3060x3060, 2700x1200, 2592x1944, 2048x1536, 1920x1080, 1600x1200, 1440x1080, 1280x960, 1280x720, 1200x1200, 1080x1080, 1024x768, 800x600, 800x480, 720x480, 640x480, 640x360, 352x288, 320x240, 176x144]
based on the requested resolution of 960x720, Camera2Session uses the nearest (in terms of total pixels) available format, namely 1024x768: Camera2Session: Using capture format: 1024x768@[7.0:30.0]

The first major reduction of resolution happens on the boundary between Java and C++:

webrtc/media/base/video_adapter.cc

Lines 234 to 236 in 7ddfc43

    
           const Fraction scale = 
        
               FindScale(*cropped_width, *cropped_height, target_pixel_count, 
        
                         max_pixel_count, variable_start_scale_factor_);

Using the requested pixel count (960x720=691200) as maximum, the video adapter tries to get the input video (1024x768=786432) below that maximum. It only uses 2/3 and 3/4 as factors for efficiency reasons. 3/4 on each axis reduces the number of overall pixels to 9/16 which is ~56%. The current resolution is now 768x576

This downscaling causes simulcast to only send 2 layers instead of 3 because it doesn't reach the 960*540 bucket anymore:

webrtc/video/config/simulcast.cc

Lines 82 to 106 in 7ddfc43

    
           constexpr const SimulcastFormat kSimulcastFormatsVP8[] = { 
        
               {1920, 1080, 3, webrtc::DataRate::KilobitsPerSec(5000), 
        
                webrtc::DataRate::KilobitsPerSec(4000), 
        
                webrtc::DataRate::KilobitsPerSec(800)}, 
        
               {1280, 720, 3, webrtc::DataRate::KilobitsPerSec(2500), 
        
                webrtc::DataRate::KilobitsPerSec(2500), 
        
                webrtc::DataRate::KilobitsPerSec(600)}, 
        
               {960, 540, 3, webrtc::DataRate::KilobitsPerSec(1200), 
        
                webrtc::DataRate::KilobitsPerSec(1200), 
        
                webrtc::DataRate::KilobitsPerSec(350)}, 
        
               {640, 360, 2, webrtc::DataRate::KilobitsPerSec(700), 
        
                webrtc::DataRate::KilobitsPerSec(500), 
        
                webrtc::DataRate::KilobitsPerSec(150)}, 
        
               {480, 270, 2, webrtc::DataRate::KilobitsPerSec(450), 
        
                webrtc::DataRate::KilobitsPerSec(350), 
        
                webrtc::DataRate::KilobitsPerSec(150)}, 
        
               {320, 180, 1, webrtc::DataRate::KilobitsPerSec(200), 
        
                webrtc::DataRate::KilobitsPerSec(150), 
        
                webrtc::DataRate::KilobitsPerSec(30)}, 
        
               // As the resolution goes down, interpolate the target and max bitrates down 
        
               // towards zero. The min bitrate is still limited at 30 kbps and the target 
        
               // and the max will be capped from below accordingly. 
        
               {0, 0, 1, webrtc::DataRate::KilobitsPerSec(0), 
        
                webrtc::DataRate::KilobitsPerSec(0), 
        
                webrtc::DataRate::KilobitsPerSec(30)}};

The simulcast layers are sorted by ascending resolution and originally have scale factors 4, 2 and 1
Finally, due to a bug in the EncoderStreamFactory, the two remaining layers (768x576 and 384*288) are further reduced to 384x288 and 192x144, because they are mapped to the scale factors 4 and 2, not 2 and 1:

webrtc/video/config/encoder_stream_factory.cc

Line 358 in 7ddfc43

layers[i].active = encoder_config.simulcast_layers[i].active;

The solution(s)

one of
- don't allow Camera2Session to take larger capture format than requested to prevent VideoAdapter to half the video stream later on
- allow the VideoAdapter a slightly increased maximum resolution, similar to max_roundup_rate in simulcast layer selection
maybe add 4:3 tables for simulcast layer selection to compensate for the lower resolution of 4:3 videos compared to 16:9 videos for same height. Currently 4:3 videos tend to fall into lower buckets (with lower layer count)
fix layer mapping bug (PR follows, EDIT here it is: fix: correctly map selected layers to simulcast layers #134)

one more thing

With and without the bug fix, livekit server reports a "Good" connection quality for that phone despite being on the same network. On a different phone, that has 960x720 in its native capture formats, layer selection works and connection quality is "Excellent". This is likely a bug in livekits scorer that I will investigate separately. I still wanted to mention it here because it was the reason for my investigations

The text was updated successfully, but these errors were encountered:

this partly fixes webrtc-sdk#133. Quick summary: - on lower resolutions, simulcast only sends 2 (or even 1) layer. - The layers are ordered from low to high resolution (downscale factors 4, 2, 1) - previously, when only sending 2 layers (factors 2, 1), the layer with factor 2 was mapped to the layer with factor 4, i.e. scaled down again) - with this patch, the layers are mapped to the correct factor and not scaled down again

davidliu · 2024-07-17T09:21:22Z

Hey, thanks for the report and PR! I'll take a look at this over soon.

davidliu · 2024-07-17T09:50:23Z

With and without the bug fix, livekit server reports a "Good" connection quality for that phone despite being on the same network. On a different phone, that has 960x720 in its native capture formats, layer selection works and connection quality is "Excellent". This is likely a bug in livekits scorer that I will investigate separately. I still wanted to mention it here because it was the reason for my investigations

We haven't yet looked into this deeply, but someone mentioned that because the server expected three layers and got only two, the score could be dinged.

holzgeist · 2024-07-17T10:06:57Z

We haven't yet looked into this deeply, but someone mentioned that because the server expected three layers and got only two, the score could be dinged.

That was me after some more digging 😃

davidliu · 2024-07-19T13:00:16Z

Looks like this might be a flutter-webrtc issue, as they use the adaptOutputFormat functions with the requested resolution. This causes the initial downscaling pointed out at the Java/C++ layer when the actual capture format is larger.

@cloudwebrtc I think it might be good to just remove the adaptOutputFormat calls here, or at least adding an option to skip the adaptOutputFormat call so that frames are sent to the native layer unadapted (the capturer should be capturing close to the target anyways). Alternatively, we could bump in the values passed into adaptOutputFormat by some factor (say 1.5) to allow for the rounding up of the camera capture format. Android SDK doesn't use this method, so should be fine to remove though.
4:3 buckets probably not needed, since the Livekit sdks determine by the longest side whether we will use 3 layers or not (which means 4:3 will have more pixels than the 16:9 with the same longest side), and we require at least 960 on the longest side, which lines up with the WebRTC buckets.
I think we actually can't use the PR, since this affects the layer ordering information and the SFUs are sensitive to that. However, I think fixing up the flutter-webrtc code to accurately get the actual video camera dimensions used will address the issue of simulcast layer limiting and thus make this not needed.

There's another alternative we can use (which also addresses point 2), which is to simply turn off the native simulcast layer limiting so that we don't run into this mismatch in the first place.

@cloudwebrtc @hiroshihorie we can turn off the simulcast layer limiting by adding WebRTC-LegacySimulcastLayerLimit/Disabled/ as a field trial. Note, for smaller resolutions (under 480px longest side), the native code limited to 1 layer only, and we should similarly follow suit and set a minimum longest side to send 2 layers regardless of whether we go with disabling the field trial or not.

holzgeist · 2024-07-22T08:50:20Z

Hi @davidliu
thanks for your thorough review and quick fixes in the linked PRs. I tested them and they fix all the described problems in my bug report above ❤️

Even though it's not needed anymore, can you elaborate on the layer ordering? I don't see where I change it. The resulting layer array is the same as before, but has correct resolutions/scaling factors set.

davidliu · 2024-07-22T09:33:01Z

@holzgeist looking at it again, I think I just brain-farted and thought it reversed the order. The ordering should be fine. I think we can actually use it actually. I'll need to do a little more digging and testing though

holzgeist · 2024-07-22T11:31:38Z

@davidliu thanks :)
I'll close the issue for now though, because the problems are fixed and the PRs merged. Let's continue discussion on PR #134 (if needed)

This was referenced Jul 15, 2024

fix: correctly map selected layers to simulcast layers #134

Open

[bug] Too low quality score livekit/client-sdk-flutter#557

Closed

This was referenced Jul 22, 2024

Report actual sizes for camera media stream track and remove adaptOutputFormat usage flutter-webrtc/flutter-webrtc#1636

Merged

fix: use getSettings() to get actual dimensions for mobile livekit/client-sdk-flutter#560

Merged

holzgeist closed this as completed Jul 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Low resolution on some Android phones #133

Low resolution on some Android phones #133

holzgeist commented Jul 15, 2024 •

edited

Loading

davidliu commented Jul 17, 2024

davidliu commented Jul 17, 2024

holzgeist commented Jul 17, 2024

davidliu commented Jul 19, 2024 •

edited

Loading

holzgeist commented Jul 22, 2024

davidliu commented Jul 22, 2024

holzgeist commented Jul 22, 2024

Low resolution on some Android phones #133

Low resolution on some Android phones #133

Comments

holzgeist commented Jul 15, 2024 • edited Loading

The issue

The deep dive into WebRTC

The solution(s)

one more thing

davidliu commented Jul 17, 2024

davidliu commented Jul 17, 2024

holzgeist commented Jul 17, 2024

davidliu commented Jul 19, 2024 • edited Loading

holzgeist commented Jul 22, 2024

davidliu commented Jul 22, 2024

holzgeist commented Jul 22, 2024

holzgeist commented Jul 15, 2024 •

edited

Loading

davidliu commented Jul 19, 2024 •

edited

Loading