Non-Linear Stretch (to Full Screen)

[sevdestruct]

stretch game to fullscreen 16:9. ( with aspect ratio stretched to fill all screen)
(original request extracted from comment for 'per-core CRT toggles request' (https://github.com/jasarien/Provenance/issues/634) made by @larssteenhoff)

Full Request (with non-linear addition):
Setting option stretch to full screen (as in fitting to full width of a display). Highly recommend doing this as non-linear stretch so it maintains the integrity of the art on screen, centrally, maximizing the screen equity without making all the characters look fat/wide as linear stretch would do uniformly.

[larssteenhoff]

Ideally both linear and non linear strech are supported.

For example I play f-zero many times full fullscreen stretch and yes there is distortion but for this game i dont care because its a car.

same for many other games. a custom aspect ration pixel value can be used too. this would accomodate both options.

[larssteenhoff]

video reference:

https://youtu.be/t7_NSTJ1ZX0

[sevdestruct]

Mocked up a sample of Chrono Trigger using a warp to simulate a non-linear stretch (8:7 -> 16:9):

Scaled:

Stretched - Linear:

Stretched - Non-linear:

I think it tends to be a decent compromise, to push the offensive stretches to your peripheral. Thoughts?

[sevdestruct]

Best example I know of of Non-Linear Stretch is from Plex Media Player. Here are some snaps to illustrate what I mean. Fatness effect of linear stretch is worse with human characters, of course, but Teenage Mutant Ninja Turtles will be good enough to illustrate the point.

The linear stretch tends to be more offensive when starting from something more narrow (like the SNES ratio of 8:7 versus the 4:3 ratio of the show below)…

Scaled:

Stretch (Linear):

Stretch (Non-linear):

---

Scaled:

Stretch (Linear):

Stretch (Non-linear):

---

Scaled:

Stretch (Linear):

Stretch (Non-linear):

---

Stretch (Linear):

Stretch (Non-linear):

[larssteenhoff]

I see what you mean, I'm just thinking out loud bit how will this look in a game like mario, were the screen scrolls all the time from right to left, won't the enemies be warped/distored when moving closer to the middle of the screen? I can image this to be a bit strange. Personally I prefer fixed linear scaled aspect ratio, but choice is a good thing!

[sevdestruct]

@larssteenhoff: True, you might see it, but it's variable / gradual - you'll see it if you're looking for it - I could do a quick mockup of that later. Either way, I think an option for either would be great.

[braindx]

If someone can track down the math that is typically used for this non-linear warp or the corresponding code in another project I might have time to add this.

[sevdestruct]

@braindx, funny, I was just thinking i might ask you if you would be the one that could solve this considering your great work on the renderer recently and when i finally got home from work here you are posting on my issue! That would be great. To this day I have never seen an emulator of any sort offer this, and have always wondered why, however it isn't entirely foreign to video players. Would be super cool if Provenance becomes the first to apply it.

I don't know the math off hand, but I can try to sort it out with you, principally it's like applying the view to a mesh that stretches it variably increasing more the further from center, like a ramp - would be like a curve of values. In my mockups above I was simulating it by warping a frame in Photoshop screens of that warp here:

…as you can see i was trying to keep middle area respected the most, the however a warp mesh photoshop is not a perfect representation of how this would be handled, but similar in principle -- was just using this to show proof of concept -- I also found this link that shows same idea applied in After Effects, and shows the mesh distortion a lot better, or more accurately:

[JoeMatt]

libretro already has a ton of shaders, including some different scalers. Not sure if any are non-linear or if they're iOS compatible but maybe something for reference at least.

Check out, https://github.com/libretro/glsl-shaders

Another GLES 2.0 shader resource,
https://github.com/h3rb/gml-pro/tree/master/GML-Pro-Pack2.gmx/shaders
Though I don't see any scalers here, there are some that would be neat to use beyond the existing CRT shader. Would be nice to have a menu to select different shaders like RetroArch and OpenEMU have.

No idea about shaders myself but something I'm looking to try out.

[sevdestruct]

@braindx : found something else that might be of help. Kodi (media player…xbmc) seems to have implemented non-linear stretch in their app, so i was sifting through their github code: gl_stretch.glsl which might shine some light on the math if I have targeted the right thing, but it's gotta be in their code somewhere.

[sevdestruct]

Some redditing and I have discovered this is also referred to as Dynamic Stretching or Anamorphic Dynamic Stretch, as discussed on this GoPro related article, so perhaps some searching around this might turn up something… video of after effects method described

[sevdestruct]

posting this as summary from slack conversation: file i made to sort out minSegment and maxSegment of the nonLinearStretch mesh, understanding the difference of the origin mesh and the destination mesh and use the difference between the segment widths to determine the parameters.

you might want to zoom into it…

Essentially what is happening visually here is:

1. get both the pixel ratios of the core/console scaled to max vert of the view it’s in, and the ratio of the fullscreen view
2. divide those into segments as a mesh (in this case, 16 segments) and get the value of 1 segment for both, let’s call that console.scaledRatio.width.segment and view.width.segment
3. once you have that, get the difference of those two segments, ratioDifference
4. use the originating, scaled ratio segment: console.scaledRatio.width.segment as the minSegment for the non-linear stretch
5. and add the ratioDifference to view.width.segment, which becomes the maxSegment
6. missing the math in the middle.. but i could use a stepped shape blend from one to the other to replicate it..

…so, we need to figure out the ramp or curve math through the segmentation if anybody can help, or if there is better math or a different way instead to go from minimum mesh segment to curve up the maximum mesh segment as it fills to total width of the view… Then, that.

[JoeMatt]

I have a basic idea of how you'd do this, you just need to choose the transform function. Let's say you choose sin() so you have sin() 0 -> 90 -> 180 => 0...1...0 on a curve.

So now you're looking to map the new pixels to the old. Lets imagine it horizontal line by horizontal line, where the old width was 640 and you're new width is 1080, and you're at pixel X.

let oldWidth = 640
let newWidth = 1080

for int i; i=0, i<= 180{
 let passedMiddle = i > (180/2)
 var mappedPixelIndex : float = sin( (newWidth * i ) / 640 )

// Deal with the fact that sin goes back to 0 after mid point
 mappedPixelIndex = passedMiddle ? passedMiddle + (oldWidth/2) : passedMiddle
 // Now you have the index in the old line, bit's a float, so do a weighted average
 // something like, where currentLine is the buffer of pixels in the original image for that row
 let justThedecimal = mappedPixel - floor(mappedPixel)
 let newRGB =avg( currentLineRGBBuffer[floor(mappedPixelIndex)] * (1-justTheDecimal),  currentLine[ceil(mappedPixelIndex)] * justTheDecimal
 // average is some kind of functions that averages 2 RGB value, and the multiplication for the two RGB values
}

So basically each new pixel is a weighted average of where the mapping function (this case sin()) lands in the middle of an old pixel, choose the closest neighbor pixel.

Now my linear algebra is pretty terrible, this can probably be done with a matrix function and no have to do it line by linen would scale in both directions, but this is where someone with OpenGL experience and better math would come in, though you could do something like above. If the vertical scaling is just linear it's pretty easy to figure out which line you should be sampling from, just do the linear ration and floor it. Or, do the same idea where you grab two lines at do a weighted average of the 4 pixels it's nearest to by the x and y mappedIndex floats you'd get by doing the conversion. Each line you'd grab the 2nd horizontal line and do an average on top of the old average, repeat with the next line, so each line gets 2 passes basically. Though since you're mapping a larger space to a smaller space I think that's not necessary, if you were shrinking the image you'd want to do a 2D weighted average of all the pixels you're combining into the smaller space which sounds more complicated.

I haven't been in a math room in over a decade dno't beat me up that this is far from efficient and I didn't use radians

Also rather than sin you'd probably want some kind of inverse log function so the drop off isn't so quick at the edges. sin would stretch more in the middle and less on the edges, I think...or maybe it'd be relatively even proportionately, my head hurts thinking about it too much.

Also, you can probably half the iterations by not doing the middle point hack, and instead work on two pixels at a time from opposite edges of the input buffer but just subtracting the mapped value from the input length and only looping to length/2

[braindx]

Yes, that's essentially right, but you don't need to make it that complex because you're not transforming it vertically at all. You just need to define a 1D function for transforming it horizontally. So, if the transform for a given pixel is usually (inX,inY)->(outX,outY) it just becomes (inX,inY)->(nonlinear(outX),outY).

In Kodi's repo, they have this shader: https://github.com/xbmc/xbmc/blob/master/system/shaders/GL/1.5/gl_stretch.glsl

That seems wrong to me because they're just squaring the x which seems like it would result in more distortion at the center and less at the edges...

I've mocked up an example, where I've implemented a sample shader that demonstrates what XBMC's function does as well as a function I wrote that warps the opposite way. By default it shows a checkerboard and my function (alternates between nonlinear and linear stretching), but you can switch it to preview Super Mario Bros 3 or Super Metroid by changing the numbers for TEST_IMAGE at the top or change it to XBMC's function by changing USE_XBMC_NONLINEAR to 1. I think you'd want something a bit more complicated than either of those, probably one with control for how much of the screen is undistorted and then some sort of ramp that controls how quickly that escalates to more extreme distortion at the edges.

[jacobhallgren]

How is work on this going :)?

[jacobhallgren]

Any updates?

[jacobhallgren]

Anyone working on this?

[EhsanWaheed]

Any potential development on this, as more people are curious about this, with the recent test flight and soon App Store submission?