feat(howto): add a new section to introduce client-side video upscali…

…ng feature Add a new section to introduce client-side video upscaling feature provided by Anbox Streaming SDK
canonical · Jul 29, 2024 · 2c77883 · 2c77883
1 parent fb620d7
commit 2c77883
Show file tree

Hide file tree

Showing 2 changed files with 54 additions and 0 deletions.
diff --git a/howto/stream/enable-client-side-video-upscaling.md b/howto/stream/enable-client-side-video-upscaling.md
@@ -0,0 +1,53 @@
+(howto-enable-client-side-video-upscaling)=
+# How to enable client-side video upscaling
+
+The [Anbox Streaming SDK](https://github.com/canonical/anbox-streaming-sdk) offers experimental video upscaling features for client side streaming, leveraging [AMD FidelityFX Super Resolution 1.0](https://gpuopen.com/fidelityfx-superresolution/). This feature allows video to be transmitted at lower resolutions, saving bandwidth without losing too much detail. The SDK sets up a [WebGL](https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API/Tutorial/Getting_started_with_WebGL) context and utilises a fragment shader based on the FSR 1.0 algorithm for post-processing, producing clearer and sharper video frames before they are displayed on the screen.
+
+## Initiate the `AnboxStream` object
+
+To enable client-side video upscaling, initiate the `AnboxStream` object as follows:
+
+```html
+<!DOCTYPE html>
+    stream = new AnboxStream({
+        targetElement: "anbox-stream",
+        experimental: {
+            upscaling: {
+              enabled: true,
+            },
+        },
+        ...
+    });
+```
+
+### Use target frame rate
+
+The option `experimental.upscaling.useTargetFrameRate` allows canvas to use the refresh frame rate setting in the Anbox container for rendering video frames. This avoids relying on `HTMLVideoElement.requestVideoFrameCallback` function, which can occasionally be fired [one v-sync late](https://wicg.github.io/video-rvfc/#introduction). This option is off by default and requires explicit activation.
+
+```{note}
+On a WebView where the `HTMLVideoElement.requestVideoFrameCallback` function is not supported, this option will be enabled by default.
+```
+
+### Use custom fragment shader
+
+The option `options.experimental.upscaling.fragmentShader` allows the use of a custom fragment shader-based upscaling algorithm for video streaming. By providing the fragment shader source code in this option, the canvas can use the custom fragment shader for video upscaling instead of the default FSR 1.0 shader. Currently, the implementation supports only single-pass rendering, which uses a single fragment shader to perform the upscaling.
+
+When using a custom fragment shader in WebGL, you need to manage the following pre-defined variables to interact with textures and control rendering:
+
+- **uSampler**
+  Type: sampler2D
+  Description: this uniform represents the texture sampler for your fragment shader. It allows the shader to sample from the texture bound to texture unit 0.
+
+- **vTextureCoord**
+  Type: varying vec2
+  Description: This varying variable is passed from the vertex shader to the fragment shader and represents the texture coordinates for the current fragment. It is used to sample the texture at specific coordinates.
+
+- **uResolution**
+  Type: vec2
+  Description: This uniform contains the dimensions of the rendering canvas, typically set to the width and height of the canvas in pixels. It helps in normalising texture coordinates for effects like scaling and post-processing.
+
+## Related topics
+
+* {ref}`exp-application-streaming`
+* {ref}`tut-set-up-stream-client`
+* {ref}`sec-streaming-sdk`
diff --git a/howto/stream/landing.md b/howto/stream/landing.md
@@ -9,6 +9,7 @@ The following guides in this section describe how to use the Streaming Stack to
 Access stream gateway <access-stream-gateway>
 Exchange out-of-band-data <exchange-oob-data>
 Integrate client side keyboard <integrate-virtual-keyboard>
+Enable client-side video upscaling <enable-client-side-video-upscaling>
 ```
 
 See {ref}`exp-application-streaming` for an introduction to the Streaming Stack.