Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HEVC Main10 cross architecture/software incompatiblity #10

Open
bmegli opened this issue Sep 22, 2020 · 6 comments
Open

HEVC Main10 cross architecture/software incompatiblity #10

bmegli opened this issue Sep 22, 2020 · 6 comments
Labels
bug Something isn't working

Comments

@bmegli
Copy link
Owner

bmegli commented Sep 22, 2020 •
edited
Loading

Testing between NHVE and unhvd-native

Works:

  • KabyLake (Ubuntu 18.04, FFmpeg 3.4, i965) -> KabyLake (Ubuntu 18.04, FFmpeg 3.4, i965)
  • IceLake (Ubuntu 20.04, FFmpeg 4.2, iHD) -> - IceLake (Ubuntu 20.04, FFmpeg 4.2, iHD)
  • IceLake (Ubuntu 20.04, FFmpeg 4.2, iHD) -> KabyLake (Ubuntu 18.04, FFmpeg 3.4, i965)
  • KabyLake((Ubuntu 20.04, FFmpeg 4.2, i965) -> IceLake (Ubuntu 20.04, FFmpeg 4.2, iHD)

Doesn't work:

  • KabyLake (Ubuntu 18.04, FFmpeg 3.4, i965) -> IceLake (Ubuntu 20.04, FFmpeg 4.2, iHD)

Debug output from FFMpeg:

[hevc @ 0x5558e875d480] Failed to end picture decode issue: 23 (internal decoding error).
[hevc @ 0x5558e875d480] hardware accelerator failed to decode picture
@bmegli bmegli added the bug Something isn't working label Sep 22, 2020
@bmegli
Copy link
Owner Author

bmegli commented Sep 22, 2020 •
edited
Loading

Scenario which doesn't work:

  • KabyLake (Ubuntu 18.04, FFmpeg 3.4, i965) -> IceLake (Ubuntu 20.04, FFmpeg 4.2, iHD)

Suspicious log elements from decoder:

[AVHWDeviceContext @ 0x5558e87b1240] Format 0x41524742 -> bgra.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x42475241 -> argb.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x41424752 -> rgba.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x52474241 -> abgr.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x58524742 -> bgr0.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x42475258 -> 0rgb.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x58424752 -> rgb0.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x52474258 -> 0bgr.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x30335241 -> unknown.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x30334241 -> unknown.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x30335258 -> unknown.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x30334258 -> unknown.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x36314752 -> unknown.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x56555941 -> unknown.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x30303859 -> gray.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x3231564e -> nv12.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x3132564e -> unknown.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x32595559 -> yuyv422.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x59565955 -> uyvy422.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x32315659 -> yuv420p.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x30323449 -> yuv420p.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x50313134 -> yuv411p.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x48323234 -> yuv422p.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x56323234 -> yuv440p.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x50343434 -> yuv444p.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x33434d49 -> unknown.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x30313050 -> p010le.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x30313259 -> unknown.
[AVHWDeviceContext @ 0x5558e87b1240] Format 0x30313459 -> unknown.
#...
[hevc @ 0x5558e875d480] nal_unit_type: 32(VPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x5558e875d480] nal_unit_type: 33(SPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x5558e875d480] nal_unit_type: 34(PPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x5558e875d480] nal_unit_type: 19(IDR_W_RADL), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x5558e875d480] Decoding VPS
[hevc @ 0x5558e875d480] Main profile bitstream
[hevc @ 0x5558e875d480] Decoding SPS
[hevc @ 0x5558e875d480] Main profile bitstream
[hevc @ 0x5558e875d480] Decoding VUI
[hevc @ 0x5558e875d480] Decoding PPS
[hevc @ 0x5558e875d480] Format vaapi_vld chosen by get_format().
[hevc @ 0x5558e875d480] Format vaapi_vld requires hwaccel initialisation.
[hevc @ 0x5558e875d480] Considering format 0x3231564e -> nv12.
[hevc @ 0x5558e875d480] Picked nv12 (0x3231564e) as best match for yuv420p10le.
#...
[hevc @ 0x5558e875d480] Failed to end picture decode issue: 23 (internal decoding error).
[hevc @ 0x5558e875d480] hardware accelerator failed to decode picture

There are 3 suspicious elements here:

  • both Main10 and p010le specified manually for decoder
  • nv12 is not a good match for yuv420p10le (8 bit depth matched for input 10 bit depth, why?)
  • there is reported format that would match almost ideally (p010le)
  • Main profile bitstream (instead of Main10 profile bitstream, why?)

@bmegli
Copy link
Owner Author

bmegli commented Sep 22, 2020 •
edited
Loading

Comparing:

  • not working KabyLake (Ubuntu 18.04, FFmpeg 3.4, i965) -> IceLake (Ubuntu 20.04, FFmpeg 4.2, iHD)
  • working IceLake (Ubuntu 20.04, FFmpeg 4.2, iHD) -> - IceLake (Ubuntu 20.04, FFmpeg 4.2, iHD)

Same output from log:

# Same supported formats
# ...
[hevc @ 0x5651a3e12480] nal_unit_type: 32(VPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x5651a3e12480] nal_unit_type: 33(SPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x5651a3e12480] nal_unit_type: 34(PPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x5651a3e12480] nal_unit_type: 19(IDR_W_RADL), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x5651a3e12480] Decoding VPS
[hevc @ 0x5651a3e12480] Main 10 profile bitstream
[hevc @ 0x5651a3e12480] Decoding SPS
[hevc @ 0x5651a3e12480] Main 10 profile bitstream
[hevc @ 0x5651a3e12480] Decoding VUI
[hevc @ 0x5651a3e12480] Decoding PPS
[hevc @ 0x5651a3e12480] Format vaapi_vld chosen by get_format().
[hevc @ 0x5651a3e12480] Format vaapi_vld requires hwaccel initialisation.
[hevc @ 0x5651a3e12480] Considering format 0x30313050 -> p010le.
[hevc @ 0x5651a3e12480] Picked p010le (0x30313050) as best match for yuv420p10le.
# ...
# decoding success
  • both Main10 and p010le specified manually for decoder
  • yuv420p10le mapped correctly to p010le (why?)
  • Main 10 profile bitstream (ok)

@bmegli
Copy link
Owner Author

bmegli commented Sep 22, 2020 •
edited
Loading

Comparing:

  • not working KabyLake (Ubuntu 18.04, FFmpeg 3.4, i965) -> IceLake (Ubuntu 20.04, FFmpeg 4.2, iHD)
  • working KabyLake (Ubuntu 18.04, FFmpeg 3.4, i965) -> KabyLake (Ubuntu 18.04, FFmpeg 3.4, i965)

Same output from log:

[AVHWDeviceContext @ 0x562c0de8f7a0] Format 0x32315659 -> yuv420p.
[AVHWDeviceContext @ 0x562c0de8f7a0] Format 0x30323449 -> unknown.
[AVHWDeviceContext @ 0x562c0de8f7a0] Format 0x3231564e -> nv12.
[AVHWDeviceContext @ 0x562c0de8f7a0] Format 0x32595559 -> yuyv422.
[AVHWDeviceContext @ 0x562c0de8f7a0] Format 0x59565955 -> uyvy422.
[AVHWDeviceContext @ 0x562c0de8f7a0] Format 0x48323234 -> yuv422p.
[AVHWDeviceContext @ 0x562c0de8f7a0] Format 0x58424752 -> rgb0.
[AVHWDeviceContext @ 0x562c0de8f7a0] Format 0x58524742 -> bgr0.
[AVHWDeviceContext @ 0x562c0de8f7a0] Format 0x30313050 -> p010le.

[hevc @ 0x562c0de8be40] nal_unit_type: 32(VPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x562c0de8be40] nal_unit_type: 33(SPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x562c0de8be40] nal_unit_type: 34(PPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x562c0de8be40] nal_unit_type: 19(IDR_W_RADL), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x562c0de8be40] Decoding VPS
[hevc @ 0x562c0de8be40] Main profile bitstream
[hevc @ 0x562c0de8be40] Decoding SPS
[hevc @ 0x562c0de8be40] Main profile bitstream
[hevc @ 0x562c0de8be40] Decoding VUI
[hevc @ 0x562c0de8be40] Decoding PPS
[AVHWFramesContext @ 0x7ffad4012360] Created surface 0x4000000.
[AVHWFramesContext @ 0x7ffad4012360] Direct mapping possible.
# ...
# decoding success
  • both Main10 and p010le specified manually for decoder
  • Main profile bitstream reported (carrying 10 bit data, should not happen?)

@bmegli
Copy link
Owner Author

bmegli commented Sep 22, 2020

Comparing:

  • not working KabyLake (Ubuntu 18.04, FFmpeg 3.4, i965) -> IceLake (Ubuntu 20.04, FFmpeg 4.2, iHD)
  • working KabyLake (Ubuntu 20.04, FFmpeg 4.2, i965) -> IceLake (Ubuntu 20.04, FFmpeg 4.2, iHD)

Same output from log:

# Same supported formats
# ...
hevc @ 0x563ccf402480] nal_unit_type: 33(SPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x563ccf402480] nal_unit_type: 34(PPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x563ccf402480] nal_unit_type: 19(IDR_W_RADL), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x563ccf402480] Decoding VPS
[hevc @ 0x563ccf402480] Main 10 profile bitstream
[hevc @ 0x563ccf402480] Decoding SPS
[hevc @ 0x563ccf402480] Main 10 profile bitstream
[hevc @ 0x563ccf402480] Decoding VUI
[hevc @ 0x563ccf402480] Decoding PPS
[hevc @ 0x563ccf402480] Format vaapi_vld chosen by get_format().
[hevc @ 0x563ccf402480] Format vaapi_vld requires hwaccel initialisation.
[hevc @ 0x563ccf402480] Considering format 0x30313050 -> p010le.
[hevc @ 0x563ccf402480] Picked p010le (0x30313050) as best match for yuv420p10le.
[AVHWFramesContext @ 0x7ff0bc0044c0] Created surface 0.
[AVHWFramesContext @ 0x7ff0bc0044c0] Direct mapping possible.
# ...
# decoding success
  • both Main10 and p010le specified manually for decoder (but it doesn't seem to impact anything)
  • Main10 bitstream correctly reported

We infer that:

  • bitstream profile is not determined by hardware
    • same KabyLake with 18.04 streaming results in Main and problem on receiving side
    • so it is determined by driver/libva/FFmpeg
  • the new pipeline (in 20.04) is more picky to what is in the bitstream

@bmegli
Copy link
Owner Author

bmegli commented Sep 22, 2020

Takeaways:

  • 20.04 pipeline is compatible with 20.04 pipeline (regardless of hardware encoder generation).
  • 18.04 can receive 20.04
  • 20.04 can't receive 18.04

18.04 -> 20.04 problem lies in driver/libva/FFmpeg layer but it is possible that the current strict behaviour is correct (?).

@bmegli
Copy link
Owner Author

bmegli commented Sep 23, 2020 •
edited
Loading

When streaming from 18.04 to 20.04 Main (not Main10) profile is detected.

The H.265 standard, page 249 defines general_profile_idc and general_profile_compatibility_flag values for HEVC Main10.

Conformance of a bitstream to the Main 10 profile is indicated by general_profile_idc being equal to 2 or general_profile_compatibility_flag[2] being equal to 1

Analyzing 18.04 pipeline output (FFmpeg 3.4 with corresponding libva and driver versions) with HEVCESBrowser we get:

general_profile_idc = 1
general_profile_compatibility_flag[i] = { 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }

which indeed is HEVC Main profile, not HEVC Main10.

So the output, at least in marking profile, is incorrect for 18.04.

Despite pixel format correctly detected as 10 bit per pixel, 20.04 pipeline (FFmpeg 4.2, etc) matches it with 8 bpp NV12, probably due to it supposedly being in Main profile.

@bmegli bmegli mentioned this issue Sep 24, 2020
Open
@bmegli bmegli mentioned this issue Jan 6, 2021
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@bmegli