[Proposal] Implement DASH Thumbnail tracks

[peaBerberian]

Overview

This is a feature proposal to add support for DASH thumbnail tracks as specified in the DASH-IF IOP 4.3 6.2.6.

Those thumbnail tracks generally allow to provide previews when seeking, and it has been linked as such in our demo page.

In a DASH MPD

In a DASH MPD (its manifest file), such tracks are set as regular AdaptationSet, with a contentType attribute set to "image" and a specific EssentialProperty element.

To support multiple thumbnail qualities (e.g. bigger or smaller thumbnails depending on the UI, the usage etc.), multiple Representation are also possible.

A curiosity is that unlike for "trickmode" tracks (which also fill the role of optionally providing thumbnail previews in the RxPlayer, through our experimental VideoThumbnailLoader tool), thumbnail tracks are not linked to any video AdaptationSet.

So if there's multiple video tracks with different content in it, I'm not sure of how we may be able to choose the right thumbnail track, nor how to communicate it through the API.
I guess it could be communicated through a Subset element, as defined in the DASH specification to force usage of specific AdaptationSets together, but I never actually encountered this element in the wild and it doesn't seem to be supported by any player.

The API

Simple solution from other players

For the API, I saw that most other players do very few things. They generally just synchronously return the metadata on a thumbnail corresponding to a specified timestamp.

That metadata includes the thumbnail's URL (e.g. to a jpeg), height and width, but also x and y coordinates as thumbnails are often in image sprites (images including multiple images). It is then the role of the application/UI to load and crop this correctly.

This seems acceptable to me, after all UI developers are generally experienced working with images and browsers are also very efficient with it (e.g. doing an <img>.src = url vs fetching the jpeg through a fetch request + linking the content to the DOM), but I did want to explore another way for multiple reasons:

1. As the core of the RxPlayer may run in another thread (in what we call "multithreading mode"), and as for now precize manifest information is only available in the WebWorker, we would either have to make such kind of API asynchronous (which makes it much less easy to handle for an application), or to send back the corresponding metadata to main thread (with thus supplementary synchronization complexities).

2. As the thumbnail track is just another AdaptationSet/Representation in the MPD, it may be impacted in the same way by other MPD elements and attributes, like multiple CDNs, content steering...

   Though thumbnail tracks are much less critical (and they also seem explicitely more limited by the DASH-IF IOP than other media types), I have less confidence on being able to provide a stable API in which the RxPlayer would provide all necessary metadata to the application so it can load and render thumbnails, than just do the loading and thumbnail rendering ourselves.

Solution I propose

So I propose here two APIs:

/**
 * Get synchronously thumbnail information for the specified time, or
 * `null` if there's no thumbnail information for that time.
 *
 * The returned metadata does not allow an application to load and
 * render thumbnails, it is mainly meant for an application to check if
 * thumbnails are available at a particular time and which qualities if
 * there's multiple ones.
 */
getThumbnailMetadata({ time }: { time: number }): IThumbnailMetadata[] | null;

/** Information returned by the `getThumbnailMetadata` method. */
export interface IThumbnailMetadata {
  /** Identifier identifying a particular thumbnail track. */
  id: string;
  /**
   * Width in pixels of the individual thumbnails available in that
   * thumbnail track.
   */
  width: number | undefined;
  /**
   * Height in pixels of the individual thumbnails available in that
   * thumbnail track.
   */
  height: number | undefined;
  /**
   * Expected mime-type of the images in that thumbnail track (e.g.
   * `image/jpeg` or `image/png`.
   */
  mimeType: string | undefined;
}

Though with that API, it means that an application continuously has to check if there's thumbnail at each timestamp by calling again and again getThumbnailMetadata e.g. as a user moves its mouse on top of the seeking bar. So I'm still unsure with that part, we could also communicate like audio and video tracks per Period and only once.

And more importantly the loading and rendering API:

/**
 * Render inside the given `container` the thumbnail corresponding to the
 * given time.
 *
 * If no thumbnail is available at that time or if the RxPlayer does not succeed
 * to load or render it, reject the corresponding Promise and remove the
 * potential previous thumbnail from the container.
 *
 * If a new `renderThumbnail` call is made with the same `container` before it
 * had time to finish, the Promise is also rejected but the previous thumbnail
 * potentially found in the container is untouched.
 */
public async renderThumbnail(options: IThumbnailRenderingOptions): Promise<void>;

export interface IThumbnailRenderingOptions {
  /**
   * HTMLElement inside which the thumbnail should be displayed.
   *
   * The resulting thumbnail will fill that container if the thumbnail loading
   * and rendering operations succeeds.
   *
   * If there was already a thumbnail rendering request on that container, the
   * previous operation is cancelled.
   */
  container: HTMLElement;
  /** Position, in seconds, for which you want to provide an image thumbnail. */
  time: number;
  /**
   * If set to `true`, we'll keep the potential previous thumbnail found inside
   * the container if the current `renderThumbnail` call fail on an error.
   * We'll still replace it if the new `renderThumbnail` call succeeds (with the
   * new thumbnail).
   *
   * If set to `false`, to `undefined`, or not set, the previous thumbnail
   * potentially found inside the container will also be removed if the new
   * new `renderThumbnail` call fails.
   *
   * The default behavior (equivalent to `false`) is generally more expected, as
   * you usually don't want to provide an unrelated preview thumbnail for a
   * completely different time and prefer to display no thumbnail at all.
   */
  keepPreviousThumbnailOnError?: boolean | undefined;
  /**
   * If set, specify from which thumbnail track you want to display the
   * thumbnail from. That identifier can be obtained from the
   * `getThumbnailMetadata` call (the `id` property).
   *
   * This is mainly useful when encountering multiple thumbnail track qualities.
   */
  thumbnailTrackId?: string | undefined;
}

Basically this method checks which thumbnail to load, load it and render it inside the given element.

For now this is done by going through a Canvas element for easy cropping/resizing. I could also go through an image tag and CSS but I was unsure of how my CSS would interact with outside CSS I do not control, so I chose for now the maybe-less efficient canvas way.

As you can see in the method description and in its implementation, there's a lot of added complexities from the fact that we do not control the container element (the application is) and that we're doing the loading ourselves instead of just e.g. the browser through an image tag:

• Multiple renderThumbnail calls may be performed in a row, in which case we have to cancel the previous requests to avoid rendering thumbnails in the wrong order.

• If a new thumbnail request fails, we also have to remove the older thumbnail to avoid having stale data.

• Because there's a lot of operations which may take some (still minor) time and as often thumbnails are just present in the same image sprite than the one before, there is a tiny cache implementation which handles just that case: if the previous image sprite already contains the right data, we do not go through the RxPlayer's core code (which may be in another thread) and back.

Still, I find the corresponding usage by an application relatively simple and elegant:

rxPlayer.renderThumbnail({ time, container })
  .then(() => console.log("Thumbnail rendered!"))
  .catch((err) => {
    if (err,code !== "ABORTED") {
      console.warn("Error while loading thumbnails:", err);
    }
  );

[peaBerberian]

Note for later: It seems that in the demo page, the spinner is displayed below the thumbnail div instead of on top.

It's too late for CSS right now...