Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Proposal] Implement DASH Thumbnail tracks
Overview
========
This is a feature proposal to add support for DASH thumbnail tracks as
specified in the DASH-IF IOP 4.3 6.2.6.
Those thumbnail tracks generally allow to provide previews when seeking,
and it has been linked as such in our demo page.
In a DASH MPD
=============
In a DASH MPD (its manifest file), such tracks are set as regular
`AdaptationSet`, with an `contentType` attribute set to `"image"` and a
specific `EssentialProperty` element.
To support multiple thumbnail qualities (e.g. bigger or smaller
thumbnails depending on the UI, the usage etc.), multiple
`Representation` are also possible.
A curiosity is that unlike for "trickmode" tracks (which also fill
the role of optionally providing thumbnail previews in the RxPlayer,
through our experimental `VideoThumbnailLoader` tool), thumbnail tracks
are not linked to any video `AdaptationSet`.
So if there's multiple video tracks with different content in it, I'm
not sure of how we may be able to choose the right thumbnail track, nor
how to communicate it through the API.
I guess it could be communicated through a `Subset` element, as defined
in the DASH specification to force usage of specific AdaptationSets
together, but I never actually encountered this element in the wild and
it doesn't seem to be supported by any player.
The API
=======
Simple solution from other players
----------------------------------
For the API, I saw that most other players do very few things. They
generally just synchronously return the metadata on a thumbnail
corresponding to a specified timestamp.
That metadata includes the thumbnail's URL (e.g. to a jpeg), height and
width, but also x and y coordinates as thumbnails are often in image
sprites (images including multiple images). It is then the role of the
application/UI to load and crop this correctly.
This seems acceptable to me, after all UI developers are generally
experienced working with images and browsers are also very efficient
with it (e.g. doing an `<img>.src = url` vs fetching the jpeg through a
fetch request + linking the content to the DOM), but I did want to
explore another way for multiple reasons:
1. As the core of the RxPlayer may run in another thread (in
what we call "multithreading mode"), and as for now precize
manifest information is only available in the WebWorker, we would
either have to make such kind of API asynchronous (which makes it
much less easy to handle for an application), or to send back the
corresponding metadata to main thread (with thus supplementary
synchronization complexities).
2. As the thumbnail track is just another AdaptationSet/Representation
in the MPD, it may be impacted in the same way by other MPD
elements and attributes, like multiple CDNs, content steering...
Though thumbnail tracks are much less critical (and they also seem
explicitely more limited by the DASH-IF IOP than other media types),
I have less confidence on being able to provide a stable API in
which the RxPlayer would provide all necessary metadata to the
application so it can load and render thumbnails, than just do the
loading and thumbnail rendering ourselves.
Solution I propose
------------------
So I propose here two APIs:
```ts
/**
* Get synchronously thumbnail information for the specified time, or
* `null` if there's no thumbnail information for that time.
*
* The returned metadata does not allow an application to load and
* render thumbnails, it is mainly meant for an application to check if
* thumbnails are available at a particular time and which qualities if
* there's multiple ones.
*/
getThumbnailMetadata({ time }: { time: number }): IThumbnailMetadata[] | null;
/** Information returned by the `getThumbnailMetadata` method. */
export interface IThumbnailMetadata {
/** Identifier identifying a particular thumbnail track. */
id: string;
/**
* Width in pixels of the individual thumbnails available in that
* thumbnail track.
*/
width: number | undefined;
/**
* Height in pixels of the individual thumbnails available in that
* thumbnail track.
*/
height: number | undefined;
/**
* Expected mime-type of the images in that thumbnail track (e.g.
* `image/jpeg` or `image/png`.
*/
mimeType: string | undefined;
}
```
Though with that API, it means that an application continuously has to
check if there's thumbnail at each timestamp by calling again and again
`getThumbnailMetadata` e.g. as a user moves its mouse on top of the
seeking bar. So I'm still unsure with that part, we could also
communicate like audio and video tracks per Period and only once.
And more importantly the loading and rendering API:
```ts
/**
* Render inside the given `container` the thumbnail corresponding to the
* given time.
*
* If no thumbnail is available at that time or if the RxPlayer does not succeed
* to load or render it, reject the corresponding Promise and remove the
* potential previous thumbnail from the container.
*
* If a new `renderThumbnail` call is made with the same `container` before it
* had time to finish, the Promise is also rejected but the previous thumbnail
* potentially found in the container is untouched.
*/
public async renderThumbnail(options: IThumbnailRenderingOptions): Promise<void>;
export interface IThumbnailRenderingOptions {
/**
* HTMLElement inside which the thumbnail should be displayed.
*
* The resulting thumbnail will fill that container if the thumbnail loading
* and rendering operations succeeds.
*
* If there was already a thumbnail rendering request on that container, the
* previous operation is cancelled.
*/
container: HTMLElement;
/** Position, in seconds, for which you want to provide an image thumbnail. */
time: number;
/**
* If set to `true`, we'll keep the potential previous thumbnail found inside
* the container if the current `renderThumbnail` call fail on an error.
* We'll still replace it if the new `renderThumbnail` call succeeds (with the
* new thumbnail).
*
* If set to `false`, to `undefined`, or not set, the previous thumbnail
* potentially found inside the container will also be removed if the new
* new `renderThumbnail` call fails.
*
* The default behavior (equivalent to `false`) is generally more expected, as
* you usually don't want to provide an unrelated preview thumbnail for a
* completely different time and prefer to display no thumbnail at all.
*/
keepPreviousThumbnailOnError?: boolean | undefined;
/**
* If set, specify from which thumbnail track you want to display the
* thumbnail from. That identifier can be obtained from the
* `getThumbnailMetadata` call (the `id` property).
*
* This is mainly useful when encountering multiple thumbnail track qualities.
*/
thumbnailTrackId?: string | undefined;
}
```
Basically this method checks which thumbnail to load, load it and render
it inside the given element.
For now this is done by going through a Canvas element for easy
cropping/resizing. I could also go through an image tag and CSS but I was
unsure of how my CSS would interact with outside CSS I do not control, so
I chose for now the maybe-less efficient canvas way.
As you can see in the method description and in its implementation,
there's a lot of added complexities from the fact that we do not control
the container element (the application is) and that we're doing the
loading ourselves instead of just e.g. the browser through an image tag:
- Multiple `renderThumbnail` calls may be performed in a row, in which
case we have to cancel the previous requests to avoid rendering
thumbnails in the wrong order.
- If a new thumbnail request fails, we also have to remove the older
thumbnail to avoid having stale data.
- Because there's a lot of operations which may take some (still minor)
time and as often thumbnails are just present in the same image
sprite than the one before, there is a tiny cache implementation
which handles just that case: if the previous image sprite already
contains the right data, we do not go through the RxPlayer's core
code (which may be in another thread) and back.
Still, I find the corresponding usage by an application relatively
simple and elegant:
```js
rxPlayer.renderThumbnail({ time, container })
.then(() => console.log("Thumbnail rendered!"))
.catch((err) => {
if (err,code !== "ABORTED") {
console.warn("Error while loading thumbnails:", err);
}
);
```- Loading branch information
1 parent
0a66b5c
commit 35abeeb