Should be able to configure vmaf/xpsnr ffmpeg args (or enc-input should be applied to reference too)

[hashhar]

I have some videos 1920x1080 whose displaymatrix implies a -90 degree rotation. In order to apply VMAF to 1920x1080 instead of 1080x1920 I can remove the rotation metadata beforehand using -display_rotation 0 and then run ab-av1 with that "un-rotated" file as input.

Now I tried to get rid of the intermediate rotation step by passing --enc-input '-display_rotation=0' which causes the encode to work properly but VMAF fails with:

RUST_LOG=ab_av1=debug ~/.cargo/bin/ab-av1 crf-search -e libx265 --min-samples 2 -i 20240704_184110.mp4 --preset slow --min-vmaf 95 --enc-input '-display_rotation=0'
⠉ 00:00:00  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- (eta 0s)[2024-12-31T09:26:46Z INFO  ab_av1::command::sample_encode] encoding sample 1/2 crf 28
⠉ 00:00:00 crf 28 1/2 ---------------------------------------------------------------------------------------------------------------------------------------------------- (encoding,  eta 0s)[2024-12-31T09:26:46Z DEBUG ab_av1::ffmpeg] cmd `ffmpeg -y -display_rotation 0 -i /Users/ashhar.hasan/Downloads/LocalSend/DCIM/Camera/.ab-av1-1ZAKSAiriga6/20240704_184110.sample80+599f.mkv -c:v libx265 -g 300 -crf 28 -pix_fmt yuv420p -preset slow -an /Users/ashhar.hasan/Downloads/LocalSend/DCIM/Camera/.ab-av1-1ZAKSAiriga6/20240704_184110.sample80+599f.x265.crf28.slow.mp4`
⠉ 00:00:14 crf 28 1/2 ##########----------------------------------------------------------------------------------------------------------------------------------------- (vmaf,       eta 4m)[2024-12-31T09:27:00Z INFO  ab_av1::vmaf] vmaf 20240704_184110.sample80+599f.x265.crf28.slow.mp4 vs reference 20240704_184110.sample80+599f.mkv
[2024-12-31T09:27:00Z DEBUG ab_av1::vmaf] cmd `ffmpeg -r 25 -i /Users/ashhar.hasan/Downloads/LocalSend/DCIM/Camera/.ab-av1-1ZAKSAiriga6/20240704_184110.sample80+599f.x265.crf28.slow.mp4 -r 25 -i /Users/ashhar.hasan/Downloads/LocalSend/DCIM/Camera/.ab-av1-1ZAKSAiriga6/20240704_184110.sample80+599f.mkv -filter_complex [0:v]format=yuv420p,setpts=PTS-STARTPTS,settb=AVTB[dis];[1:v]format=yuv420p,setpts=PTS-STARTPTS,settb=AVTB[ref];[dis][ref]libvmaf=shortest=true:ts_sync_mode=nearest:n_threads=16 -an -sn -dn -f null -`
Error: ffmpeg vmaf exit code 234
----cmd-----
ffmpeg -r 25 -i /Users/ashhar.hasan/Downloads/LocalSend/DCIM/Camera/.ab-av1-1ZAKSAiriga6/20240704_184110.sample80+599f.x265.crf28.slow.mp4 -r 25 -i /Users/ashhar.hasan/Downloads/LocalSend/DCIM/Camera/.ab-av1-1ZAKSAiriga6/20240704_184110.sample80+599f.mkv -filter_complex [0:v]format=yuv420p,setpts=PTS-STARTPTS,settb=AVTB[dis];[1:v]format=yuv420p,setpts=PTS-STARTPTS,settb=AVTB[ref];[dis][ref]libvmaf=shortest=true:ts_sync_mode=nearest:n_threads=16 -an -sn -dn -f null -
---stderr---
ffmpeg version 7.1 Copyright (c) 2000-2024 the FFmpeg developers
  built with Apple clang version 16.0.0 (clang-1600.0.26.4)
  configuration: --prefix=/opt/homebrew/Cellar/ffmpeg/7.1_4 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags='-Wl,-ld_classic' --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libaribb24 --enable-libbluray --enable-libdav1d --enable-libharfbuzz --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox --enable-audiotoolbox --enable-neon
  libavutil      59. 39.100 / 59. 39.100
  libavcodec     61. 19.100 / 61. 19.100
  libavformat    61.  7.100 / 61.  7.100
  libavdevice    61.  3.100 / 61.  3.100
  libavfilter    10.  4.100 / 10.  4.100
  libswscale      8.  3.100 /  8.  3.100
  libswresample   5.  3.100 /  5.  3.100
  libpostproc    58.  3.100 / 58.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/ashhar.hasan/Downloads/LocalSend/DCIM/Camera/.ab-av1-1ZAKSAiriga6/20240704_184110.sample80+599f.x265.crf28.slow.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2mp41
    encoder         : Lavf61.7.100
    location-eng    : +22.1955+084.8481/
    location        : +22.1955+084.8481/
  Duration: 00:00:19.99, start: 0.000000, bitrate: 2196 kb/s
  Stream #0:0[0x1](eng): Video: hevc (Main) (hev1 / 0x31766568), yuv420p(tv, bt709, progressive), 1920x1080, 2192 kb/s, 29.96 fps, 29.96 tbr, 12043 tbn (default)
      Metadata:
        handler_name    : VideoHandle
        vendor_id       : [0][0][0][0]
        encoder         : Lavc61.19.100 libx265
Input #1, matroska,webm, from '/Users/ashhar.hasan/Downloads/LocalSend/DCIM/Camera/.ab-av1-1ZAKSAiriga6/20240704_184110.sample80+599f.mkv':
  Metadata:
    COM.ANDROID.VERSION: 14
    MAJOR_BRAND     : mp42
    MINOR_VERSION   : 0
    COMPATIBLE_BRANDS: isommp42
    COM.ANDROID.CAPTURE.FPS: 30.000000
    LOCATION        : +22.1955+084.8481/
    LOCATION-eng    : +22.1955+084.8481/
    ENCODER         : Lavf61.7.100
  Duration: 00:00:19.99, start: 0.000000, bitrate: 16961 kb/s
  Stream #1:0(eng): Video: h264 (High), yuv420p(tv, bt709, progressive), 1920x1080, 29.96 fps, 29.96 tbr, 1k tbn (default)
      Metadata:
        HANDLER_NAME    : VideoHandle
        VENDOR_ID       : [0][0][0][0]
        DURATION        : 00:00:19.994000000
      Side data:
        displaymatrix: rotation of -90.00 degrees
Stream mapping:
  Stream #0:0 (hevc) -> format:default
  Stream #1:0 (h264) -> format:default
  libvmaf:default -> Stream #0:0 (wrapped_avframe)
Press [q] to stop, [?] for help
[Parsed_libvmaf_6 @ 0x600001ceacb0] input width must match.
[Parsed_libvmaf_6 @ 0x600001ceacb0] input height must match.
[Parsed_libvmaf_6 @ 0x600001ceacb0] Failed to configure input pad on Parsed_libvmaf_6
[fc#0 @ 0x600001af7b60] Error reinitializing filters!
[fc#0 @ 0x600001af7b60] Task finished with error code: -22 (Invalid argument)
[fc#0 @ 0x600001af7b60] Terminating thread with return code -22 (Invalid argument)
[vost#0:0/wrapped_avframe @ 0x108c04db0] Could not open encoder before EOF
[vost#0:0/wrapped_avframe @ 0x108c04db0] Task finished with error code: -22 (Invalid argument)
[vost#0:0/wrapped_avframe @ 0x108c04db0] Terminating thread with return code -22 (Invalid argument)
[out#0/null @ 0x600001ee0000] Nothing was written into output file, because at least one of its streams received no packets.
frame=    0 fps=0.0 q=0.0 Lsize=       0KiB time=N/A bitrate=N/A speed=N/A
Conversion failed!
------------

Note the presence of displaymatrix: rotation of -90.00 degrees in the reference input but absence in the distorted file.

Is this expected?

[alexheretic]

--enc, --enc-input pass args only to the encoding ffmpeg call and not to the vmaf ffmpeg call.

--vfilter is used for both, so works better for scaling etc.

A workaround could be to remove the metadata before calling ab-av1. Or rotate using a vfilter.

We can probably add more configurable vmaf/xpsnr ffmpeg arg support too.

[hashhar]

Thanks for replying. I figured out a better option in meantime - -noautorotate flag to ffmpeg causes it to not look at file metadata which achieves same effect (as long as video was physically 1920x1080 and the displaymatrix is the only thing making it vertical) - won't help for videos which are actually 1080x1920 instead.

[hashhar]

Feel free to close if you think this isn't generally applicable. FWIW pre-processing files isn't too bad - specially because it allows inspecting files before doing the crf-search or vmaf calculations to make sure things are sane.