Poor quality upscales

[Netruk44]

I'm using the upscaling option, and I'm noticing that there are a couple of footguns in the upscale flow that can lead to upscaled movies being worse quality than the 1080p original.

1. If your quality is set too low before the upscale, you basically wind up upscaling compression artifacts.
   • A high quality (above the default of 75) should probably be recommended if the user is choosing to upscale.
2. Right now, fx-upscale chooses its own encoding bitrate, so whatever quality you've chosen in the BD_to_AVP GUI only applies to what gets fed into the upscaler.
   • I have a PR out for fx-upscale to provide a --bitrate option on the command line so you can set your own bitrate instead of it choosing one for you.
     • Add a bitrate command line argument to control bitrate of output video finnvoor/fx-upscale#20

I've made my own fork of BD_to_AVP that gives the user control of the --bitrate argument, but until the fx-upscale PR is completed it's probably premature to submit a PR for that here.
Commit: Netruk44@1118883

I'm just making this issue as a heads-up that this change might be useful to add soon (™️).

[Netruk44]

(I'm also still working on the change to BD_to_AVP, so don't take that commit as-is, it doesn't work 😉)

[cbusillo]

bitrate would only be used on the upscale (I think) so it should probably be named more appropriately. upscale-bitrate maybe?

Do you have suggestions on the interface/verbiage for the user? I'm really surprised quality 75 isn't enough. Maybe it depends on the type of video (motion, color ranges, etc)?

Maybe if AI Upscale is turned on, the quality gets disabled (and set high) and the ai upscale bitrate field is enabled. One of my goals is to make this easy to use without knowing details like that.

[cbusillo]

I noticed you already named it appropriately in source so disregard that

[Netruk44]

I think my commit should be used as a template more than anything. You would probably know how to architect it/lay it out for the user better than I do, I'm just trying to make it work for me 😛.

As for the quality, my test movie was an animated movie (Wreck-It Ralph) so it's possible that 75 quality is good enough for live-action films. But when highly detailed objects (such as character's faces) were rapidly moving around the screen in a small area (such as when the cameras are following a group of go-kart racers as they're going down a track), the 4K output was way blurrier than the 1080p input.

To be fair, when I was testing I didn't yet have a version of fx-upscale that you could set an output bitrate on. It's entirely possible quality 75 is good enough provided the upscaled output bitrate is also good enough.

[cbusillo]

If they don't add the change soon, we could use your fork of fx instead. Would you be able to use "quality" instead of bitrate in fx-upscale? That would make things easier, we wouldn't need a second parameter, we can reuse the MV-HEVC quality.

[Netruk44]

I'm not sure. fx-upscale uses Apple libraries for it's upscaler, so there are limited options for what can be specified.

There's an AVVideoQualityKey that looks like it might work, but reading the documentation shows:

A key to set the JPEG compression quality of the video.

So I suspect you need to be using a JPEG codec and not HEVC. I can give it a try if you think it's worth a shot.

[cbusillo]

I think for most people, it would be a better way to do it. Bitrate is hard, but 0-100 quality is easy. We can use bitrate if needed though. I look forward to checking this out. It may be why some people really liked the upscale and some didn't. It may depend on the type of video the way its currently setup without an output quality/bitrate.

[Netruk44]

I can give it a shot and see what happens later this afternoon.

[Netruk44]

Would it be possible for BD_to_AVP to detect the bitrate of either:

1. The original input video
2. The left/right videos

If so, maybe instead of presenting a bitrate option to the user, you might automatically calculate a bitrate for the user by multiplying the input bitrate by, say 1.5-2.0 or so to account for the increased resolution? That might be good enough, and less complexity the user needs to care about.

[cbusillo]

I believe ffprobe will easily tell me the bitrate of the MV-HEVC we are upscaling. I wouldn't know what factor to use though. there are four times the pixels, but I do not understand the compression. something like this would work. I haven't checked the key names. If you can come up with a good way to translate the input bitrate to the upscaled bitrate, I would say that is a better method than asking the user.

def upscale_file(input_path: Path) -> None:
    probe = ffmpeg.probe(str(input_path))
    bitrate = int(probe["streams"][0]["bit_rate"]) // 1000

[cbusillo]

We have a small group of users in a Reddit chat. If you are interested and have a Reddit user. I can invite you. I think your keep awake idea was great. It doesn't affect me since I manually sleep my Mac while plugged in, but of course that is an edge case. The upscale is another thing I wouldn't have noticed, and if we don't know it is a problem, we can't fix it.

[Netruk44]

The more I think about it, the more I think this bitrate calculation should probably live in fx-upscale and not in your project. That way everyone who tries to upscale video benefits from it.

So maybe there's no issue here at all 😛.

I don't know what the proper scale factor would be, either. I tend to just pick numbers and try them out to see what the result is like, but that's far from scientific, haha.

I'm also not a video encoding expert, so any advice I give would just be guesses based on my limited personal experience trying to upscale this one movie I've done so far.

I have a Reddit account, but I try not to be on there much. I'm more active on Discord if you have one of those.

[Netruk44]

(For what it's worth, ChatGPT suggested the scale factor be based off the number of pixels being encoded per frame. I sanity checked the calculation and it seemed fine, so with that in mind...)

I've pushed a new branch to my fork of fx-upscale: https://github.com/Netruk44/fx-upscale/tree/auto-bitrate

This version of fx-upscale automatically sets the compression bitrate based on the input bitrate, input resolution, and output resolution, if you haven't specified a bitrate on the command line already.

This means it should work as a drop-in improvement without any changes needed in this project (aside from using my fork of the fx-upscale repository).

This also means that the upscaled output video file is directly based on the quality number you set in the BD_to_AVP GUI.

The upscaled output video files will be larger, but I don't yet know how much larger. I'm doing a test run with quality set to 75 to see how things go. Just thought I would drop in to give a status update 😄

[cbusillo]

Great. Let me know how the encode goes with sizing and bitrates. I'll just have to figure out packaging with it not being on pypi.

[Netruk44]

Video	File Size	Video Bitrate
Input MKV	25.83GB	-
Left/Right Videos	15.5 GB (each)	20.77 Mbit/s
MV-HEVC (75 Quality)	10.29 GB	13.54 Mbit/s
Upscaled	41.13 GB	54.17 Mbit/s
Final output	56.87 GB	54.17 Mbit/s

(Note: Final output total bitrate (including audio) is 74.9 Mbit/s)

Final video output with this process is much, much cleaner than with the current version of dx-upscale.

75 quality is good enough to upscale, I think. Depending on your space requirements, you might want to go even lower (57 GB is a big file to move around).

For this particular movie, I think a quality around 80-85 might be more ideal, just so the 1080p MV-HEVC video's bitrate isn't lower than the left/right videos. But I don't know if I really want a larger video than this, haha.

[Netruk44]

Honestly, I think the upscaler encoding at 4x the bitrate might be a little overkill. Mathematically it might be optimal, but the user probably won't like dealing with the 60 GB video files it produces 😛.

Just to throw an idea out there, maybe there could be a command line argument on fx_upscale like --bitrate_scaling_factor where 1.0 would be 4x the bitrate and 0.5 would be 2x the bitrate.

I might try re-upscaling with the bitrate set to 40 Mbit/s (0.75 scaling factor) and 27 Mbit/s (0.5 scaling factor) to see how the output compares.

[Netruk44]

Video	File Size	Video Bitrate
2x Bitrate	20.57 GB	27.09 Mbit/s
3x Bitrate	30.84 GB	40.62 Mbit/s
4x Bitrate	41.13 GB	54.17 Mbit/s

To compare these visually, I looked at 3 different, specific frames of the movie and did a visual comparison between them (copying the frame to a file and quickly tabbing between Preview windows of the 3 different bitrates trying to see differences).

2x bitrate: If I use the Mac's built in zoom tool, I can see that the edges between some low-contrast colors (for example, purple to brown) is very slightly blurrier compared to the 4x bitrate. But that's about it. Without the assistance of the zoom tool, it's pretty indistinguishable from 4x bitrate.

3x bitrate: Actually indistinguishable from 4x the bitrate, even with the help of zooming in.

What do you think about adding the bitrate_scaling_factor to fx-upscale, and adding a Upscale Bitrate Factor spinner to BD_to_AVP that goes from ~0.1 to 1.0 (with a default of 0.5)?

(Or we could even change the scaling factor so that 1.0 is 1x the bitrate of the input, and 4.0 is 4x, and recommending a value between 2.0 and 3.0. This way it would make more sense from the user perspective)

[cbusillo]

The left right bitrate is a static value I picked. The 20Mb/s should always be better than the 35Mb/s bitrate of the source (h264 vs h265). I let the MV-HEVC quality handle the actual bitrate to be output. Basically ignore the left right bitrate and look at the source in the iso or mkv or h264 file. Also remember that before the left right files the video is h264 and after it is in h265.

I asked the spatial media kit tool creator if he could accept streams in addition to files but haven't received an answer. I added color depth to his program, but accepting streams went over my head. That would eliminate the extra conversation, space usage, and possibly speed it up.

I'll look at the rest of your info in the morning.

[cbusillo]

I just realized it's from brew not pypi. lol. I'll see if I can get it compiled and included.

[cbusillo]

I like the scaling factor method you mentioned. It would align well with the quality we are already using. My thought here is we apply the currently quality to the fx-upscale scaling factor. MV-HEVC of 75 becomes a fx-upscale scaling_factor of 0.75. If this works for you or you have a better idea, just let me know. I would then only need to modify the upscale_file function to something like this.

def upscale_file(input_path: Path) -> None:
    upscale_command = [
        config.FX_UPSCALE_PATH,
        "--bitrate_scaling_factor",
        config.mv_hevc_quality / 100,
        input_path,
    ]
    run_command(upscale_command, "Upscale video with FX Upscale plugin.")

[Netruk44]

Yes, I think the scaling_factor could be coupled with quality to keep things simple. Although for personal usage, I might make a fork of the UI to control the two numbers separately.

[Netruk44]

I've pushed an update to my auto-bitrate branch that adds a --bitrate-scaling-factor argument (note that it uses - instead of _ for the spaces).

https://github.com/Netruk44/fx-upscale/tree/auto-bitrate

(I implemented the version where scaling factor 1.0 = 4x the bitrate, so you can pass in quality / 100 as you wrote)

[cbusillo]

Ok, to keep from having separate versions I added it to the interface. Let me know what you think, test encode works and looks good.

Edit: It is building now.

v0.2.126

[Netruk44]

Oh, sorry, I think it's fine to have the scale factor linked to the quality number for most people.

I'm just weird and would like a little extra knob to tweak when I'm doing my upscales 😛

[cbusillo]

Go ahead and checkout the latest and let me know if you have any suggestions. It just finished building and posting.

[Samckatz]

Any suggestions what would be the perfect settings for Hvec quality and for Upscale quality

[Netruk44]

Looks good!

I only have one thought, but nothing major. It's a little weird you can set the upscale quality if you don't have the upscale checkbox checked, but I'm not sure how you'd handle the linked SpinBoxes in that case. Maybe you could hide the upscale spinbox? I don't know how difficult that would be to do, though.

Side note:

I'm unsatisfied with using Files preview to play these videos. You don't get room dimming or the lighting effects you normally do when you play videos through the TV app.

I just put together (with generous help from ChatGPT) as simple of a video playing app as possible and put that into a new repository. It does nothing other than open a file browser, let you pick a file, then plays the video using the standard video controls.

I aim to put this into the store for free eventually, but if you're interested in building it and trying it out for yourself, feel free.

[cbusillo]

@Netruk44 there are a few (one is soon to be released) apps that allow you to play MV-HEVC now. I think two have plex integration which is how I play my videos. Screenlit is the one I am using. You can message jnorris441 on reddit and ask for a TestFlight if you are interested.

I added a hide function when upscale is not selected. Its building and publishing now.

[cbusillo]

I also moved the quality options below the checkboxes. Between hiding the quality spin box and the audio bitrate spin box, it was annoying that the stuff would move. I'm not any good with interfaces, so this is the best I have. lol

[Netruk44]

@cbusillo Awesome, thanks for the heads up! I'd love to not have to own a video player, haha.

Maybe you could have a list of recommended ones somewhere in this repo or in the app? Would help discoverability, the app store search doesn't always work well.

[cbusillo]

So far Screenlit is the only one I have experience with and its not released.
@Samckatz might remember the other ones mentioned.

[Netruk44]

Any suggestions what would be the perfect settings for Hvec quality and for Upscale quality

I think the "perfect" setting depends on the movie and how much storage space you'd like for the movie to take up. It's more of a personal preference kind of thing.

I would recommend sticking to 75/75 from my own personal experience. That'll give you a ~40 GB video file.

If the upscaled output results in a file that's too big for your preference, crank down the upscale quality to 50, which should drop the output video size to ~30 GB, but will start to introduce compression artifacts. They may not be noticable in the AVP, I haven't tested yet.

(Edit: To further reduce file size, you can also turn down HEVC quality by a little. But not by a lot, you want the 1080p version to be clean for the upscale.)

Upscale quality can go lower than 50, too, depending on how much you notice the compression artifacts.

If you're upscaling, I personally would not move the HEVC quality setting above 75. It has a drastic effect on the final size of the upscaled video. Quality 100/100 would probably give you a >500 GB video at the end of it, for practically no discernible visual differences.

[Netruk44]

After having some time to play around with this, I think we can close this as fixed. The 4k output is definitely better now from what I'm seeing. Much less smearing and blurring in motion-heavy scenes, with a noticeable bump in sharp details compared to the 1080p original input.

If someone wants my advice for how to set things, I can write more about the settings I've tried so far. But I think long story short, 75/75 is probably optimal. At that quality, all the details of the upscale process should be preserved in the output while keeping the file size as small as it can reasonably be (Any smaller would lose the details your processor spent time generating and/or details in the original film).

[cbusillo]

If you feel like writing a blurb for the readme, I would be appreciative. You did a great job showing the sweet spot with charts and sizing.

[Samckatz]

So far Screenlit is the only one I have experience with and its not released. @Samckatz might remember the other ones mentioned.

Reality Player is great

[Netruk44]

If you feel like writing a blurb for the readme, I would be appreciative. You did a great job showing the sweet spot with charts and sizing.

Sure I can write something up. It'll take me a little while, though. I want to check a few more quality pairings before making recommendations

[Netruk44]

@cbusillo I'm noticing a bug where --bitrate-scaling-factor 0.75 is always passed to fx-upscale no matter what you set in the UI (according to the "Output Commands" text)

[Netruk44]

I think the issue is around here:

BD_to_AVP/bd_to_avp/gui/main_window.py

Line 471 in 0c23380

config.mv_hevc_quality = self.mv_hevc_quality_spinbox.value()

config.upscale_quality is never set inside of save_config, so its value never changes from the default of 75.

[cbusillo]

Fixed in master. Thank you. I swear I added those. I must have accidentally backed them out at some point.

[cbusillo]

Merged notes.