Thumbnails: Add supplementary metadata to getAvailableThumbnailTracks

Based on #1496

Problem
-------

We're currently trying to provide a complete[1] and easy to-use API for
DASH thumbnail tracks in the RxPlayer.

Today the proposal is to have an API called `renderThumbnail`, to which
an application would just provide an HTML element and a timestamp, and
the RxPlayer would do all that's necessary to fetch the corresponding
thumbnail and display it in the corresponding element.

The API is like so:
```js
rxPlayer.renderThumbnail({ element, time })
  .then(() => console.log("The thumbnail is now rendered in the element"));
```

This works and seems to me very simple to understand.

Yet, we've known of advanced use cases where an application might not
just want to display a single thumbnail for a single position. For
example, there's very known examples where an application displays a
window of multiple thumbnails at once on the player's UI to facilitate
navigation inside the content.

To do that under the solution proposed in #1496, an application could
just call `renderThumbnail` with several `element` and `time` values.

Yet for this type of feature, what the interface would want is not really
to indicate a `time` values, it actually wants basically a list of
distinct thumbnails around/before/after a given position.

By just being able to set a `time` value, an application is blind on
which `time` value is going to lead to a different timestamp (i.e. is
the thumbnail for the `time` `11` different than the thumbnail for the
`time` `12`? Nobody - but the RxPlayer - knows).

So we have to find a solution for this

[1] By complete, I here mean that we want to be able to handle its
complexities inside the RxPlayer, to ensure complex DASH situations like
multi-CDN, retry settings for requests and so on while still allowing
all potential use cases for an application.

Solution
--------

In this solution, I experiment with a second thumbnail API,
`getAvailableThumbnailTracks` (it already exists in #1496, but its role
there was only to list the various thumbnail qualities, if there are
several size for example). As this solution build upon yet stays
compatible to #1496, I chose to open this second PR on top of that
previous one.

I profit from the fact that most standardized thumbnail implementations I
know of (BIF, DASH) seem follow the principle of having evenly-spaced
(in terms of time) thumbnails (though I do see a possibility
for that to change, e.g. to have thumbnails corresponding to "important"
scenes instead, so our implementation has to be resilient).

So here, what this commit does is to add the following properties (all
optional) to a track returned by the `getAvailableThumbnailTracks` API:

  - `start`: The initial `time` the first thumbnail of that track will
    apply to

  - `end`: The last `time` the last thumbnail of that track will
    apply to

  - thumbnailsPerSegment: Individual thumbnails may be technically part of
    "segments" containing multiple consecutive thumbnails each.

    `thumbnailsPerSegment` is the number of thumbnails each of those
    segments contain.

    For example you could have stored on the server a segment which is a
    grid of 2 x 3 (2 horizontal rows and * 3 vertical columns)
    thumbnails, which the RxPlayer will load at once then "cut" the right
    way when calling `renderThumbnail`. In that example,
    `thumbnailsPerSegment` would be set to `6` (2*3).

    Note that the last segment of a content may contain less thumbnails as
    anounced here depending on the duration of the content.

  - `segmentDuration`: The "duration" (in seconds) each segments of
    thumbnails applies to (with the exception of the last thumbnail,
    which just fills until `end`)

Then, an application should have all information needed to calculate a
`time` which correspond to a different thumbnail.

Though this solution lead to a minor issue: by letting application make
the `time` operation themselves with `start`, `end`, `segmentDuration`
and so on, there's a risk of rounding errors leading to a `time`
which does not correspond to the thumbnail wanted but the one before or
after. To me, we could just indicate in our API documentation to
application developers that they should be extra careful and may add an
epsilon (or even choose a `time` in the "middle" of thumbnails each time)
if they want that type of thumbnail list feature.

Thoughts?

demo/scripts/modules/player/index.ts

@@ -343,8 +343,7 @@ const PlayerModule = declareModule(
       },
 
       getAvailableThumbnailTracks(time: number): IThumbnailTrackInfo[] {
-        const metadata = player.getAvailableThumbnailTracks({ time });
-        return metadata ?? [];
+        return player.getAvailableThumbnailTracks({ time });
       },
 
       renderThumbnail(time: number, thumbnailTrackId: string): Promise<void> {

src/main_thread/api/public_api.ts

@@ -799,6 +799,10 @@ class Player extends EventEmitter<IPublicAPIEvent> {
         width: Math.floor(t.width / t.horizontalTiles),
         height: Math.floor(t.height / t.verticalTiles),
         mimeType: t.mimeType,
+        start: t.start,
+        end: t.end,
+        segmentDuration: t.segmentDuration,
+        thumbnailsPerSegment: t.thumbnailsPerSegment,
       };
     });
   }

src/manifest/classes/__tests__/adaptation.test.ts

@@ -51,6 +51,9 @@ const minimalRepresentationIndex: IRepresentationIndex = {
   addPredictedSegments() {
     /* noop */
   },
+  getTargetSegmentDuration() {
+    return undefined;
+  },
   _replace() {
     /* noop */
   },

src/manifest/classes/__tests__/representation.test.ts

@@ -47,6 +47,9 @@ const minimalIndex: IRepresentationIndex = {
   canBeOutOfSyncError(): true {
     return true;
   },
+  getTargetSegmentDuration() {
+    return undefined;
+  },
   _replace() {
     return;
   },

src/manifest/classes/__tests__/update_period_in_place.test.ts

@@ -174,6 +174,10 @@ function generateFakeThumbnailTrack({ id }: { id: string }) {
     width: 200,
     horizontalTiles: 5,
     verticalTiles: 3,
+    start: 0,
+    end: 100,
+    segmentDuration: 2,
+    thumbnailsPerSegment: 2,
     index: {
       _update() {
         /* noop */
@@ -1433,6 +1437,10 @@ describe("Manifest - updatePeriodInPlace", () => {
           id: "thumb-2",
           mimeType: "image/png",
           verticalTiles: 3,
+          start: 0,
+          end: 100,
+          segmentDuration: 2,
+          thumbnailsPerSegment: 2,
           width: 200,
         },
       ],
@@ -1443,6 +1451,10 @@ describe("Manifest - updatePeriodInPlace", () => {
           id: "thumb-1",
           mimeType: "image/png",
           verticalTiles: 3,
+          start: 0,
+          end: 100,
+          segmentDuration: 2,
+          thumbnailsPerSegment: 2,
           width: 200,
         },
       ],
@@ -1510,6 +1522,10 @@ describe("Manifest - updatePeriodInPlace", () => {
           id: "thumb-2",
           mimeType: "image/png",
           verticalTiles: 3,
+          start: 0,
+          end: 100,
+          segmentDuration: 2,
+          thumbnailsPerSegment: 2,
           width: 200,
         },
       ],
@@ -1520,6 +1536,10 @@ describe("Manifest - updatePeriodInPlace", () => {
           id: "thumb-1",
           mimeType: "image/png",
           verticalTiles: 3,
+          start: 0,
+          end: 100,
+          segmentDuration: 2,
+          thumbnailsPerSegment: 2,
           width: 200,
         },
       ],
@@ -1588,6 +1608,10 @@ describe("Manifest - updatePeriodInPlace", () => {
           id: "thumb-1",
           mimeType: "image/png",
           verticalTiles: 3,
+          start: 0,
+          end: 100,
+          segmentDuration: 2,
+          thumbnailsPerSegment: 2,
           width: 200,
         },
       ],
@@ -1660,6 +1684,10 @@ describe("Manifest - updatePeriodInPlace", () => {
           id: "thumb-1",
           mimeType: "image/png",
           verticalTiles: 3,
+          start: 0,
+          end: 100,
+          segmentDuration: 2,
+          thumbnailsPerSegment: 2,
           width: 200,
         },
       ],

src/manifest/classes/period.ts

@@ -145,6 +145,10 @@ export default class Period implements IPeriodMetadata {
       width: thumbnailTrack.width,
       horizontalTiles: thumbnailTrack.horizontalTiles,
       verticalTiles: thumbnailTrack.verticalTiles,
+      start: thumbnailTrack.start,
+      end: thumbnailTrack.end,
+      segmentDuration: thumbnailTrack.segmentDuration,
+      thumbnailsPerSegment: thumbnailTrack.thumbnailsPerSegment,
     }));
     this.duration = args.duration;
     this.start = args.start;
@@ -305,6 +309,10 @@ export default class Period implements IPeriodMetadata {
         width: thumbnailTrack.width,
         horizontalTiles: thumbnailTrack.horizontalTiles,
         verticalTiles: thumbnailTrack.verticalTiles,
+        start: thumbnailTrack.start,
+        end: thumbnailTrack.end,
+        segmentDuration: thumbnailTrack.segmentDuration,
+        thumbnailsPerSegment: thumbnailTrack.thumbnailsPerSegment,
       })),
     };
   }
@@ -344,4 +352,59 @@ export interface IThumbnailTrack {
    * images contained vertically in a whole loaded thumbnail resource.
    */
   verticalTiles: number;
+  /**
+   * Starting `position` the first thumbnail of this thumbnail track applies to,
+   * if known.
+   */
+  start: number | undefined;
+  /**
+   * Ending `position` the last thumbnail of this thumbnail track applies to,
+   * if known.
+   */
+  end: number | undefined;
+  /**
+   * Individual thumbnails may be technically part of "segments" containing
+   * multiple consecutive thumbnails each.
+   *
+   * `thumbnailsPerSegment` is the number of `thumbnails` each segments have.
+   *
+   * For example you could have stored on the server a segment which is a grid
+   * of 2 x 3 (2 horizontal rows and 3 vertical columns) thumbnails, which the
+   * RxPlayer will load at once then "cut" the right way when calling
+   * `renderThumbnail`. In that example, `thumbnailsPerSegment` would be set to
+   * `6` (2*3).
+   *
+   * Note that the last segment of a content may contain less thumbnails as
+   * anounced here depending on the duration of the content.
+   *
+   * You may want to rely on this information alongside `segmentDuration` to
+   * construct a list of available thumbnails and/or of available segments of
+   * thumbnails.
+   */
+  thumbnailsPerSegment: number | undefined;
+  /**
+   * When loaded, thumbnails are part of so-called "segments" which may contain
+   * either a single thumbnail or a grid of them (@see `thumbnailsPerSegment`).
+   *
+   * This `segmentDuration` property indicates a duration in seconds each
+   * segment applies to. You might then want to divide that value by
+   * `thumbnailsPerSegment` to get the duration each thumbnail applies to.
+   *
+   * Note that the last segment of a content may have a lower duration depending
+   * on the duration of the content.
+   *
+   * Set to `undefined` either the duration is unknown or if the duration
+   * depends from segment to segments.
+   *
+   * For example, with a `start` set to `10`, an `end` set to `21`, a
+   * `thumbnailsPerSegment` set to `2` and a `SegmentDuration` set to
+   * `4`, there should be 3 segments, each with 2 thumbnails:
+   * 1. A segment of 2 thumbnails for the seconds: 10-14
+   *    (The first thumbnail in that segment for 10-12, the second for 12-14)
+   * 2. A segment of 2 thumbnails for the seconds: 14-18
+   *    (The first thumbnail in that segment for 14-16, the second for 16-18)
+   * 3. A segment of 2 thumbnails for the seconds: 18-21 (the end)
+   *    (The first thumbnail in that segment for 18-20, the second for 20-21)
+   */
+  segmentDuration: number | undefined;
 }

src/manifest/classes/representation_index/static.ts

@@ -155,6 +155,24 @@ export default class StaticRepresentationIndex implements IRepresentationIndex {
     log.error("A `StaticRepresentationIndex` does not need to be initialized");
   }
 
+  /**
+   * Returns the `duration` of each segment in the context of its Manifest (i.e.
+   * as the Manifest anounces them, actual segment duration may be different due
+   * to approximations), in seconds.
+   *
+   * NOTE: we could here do a median or a mean but I chose to be lazy (and
+   * more performant) by returning the duration of the first element instead.
+   * As `isPrecize` is `false`, the rest of the code should be notified that
+   * this is only an approximation.
+   * @returns {number}
+   */
+  getTargetSegmentDuration(): { duration: number; isPrecize: boolean } | undefined {
+    return {
+      duration: Number.MAX_VALUE,
+      isPrecize: false,
+    };
+  }
+
   addPredictedSegments(): void {
     log.warn("Cannot add predicted segments to a `StaticRepresentationIndex`");
   }

src/manifest/classes/representation_index/types.ts

@@ -429,6 +429,29 @@ export interface IRepresentationIndex {
    */
   initialize(segmentList: ISegmentInformation[]): void;
 
+  /**
+   * Returns an approximate for the duration of that `RepresentationIndex`s
+   * segments, in seconds in the context of its Manifest (i.e. as the Manifest
+   * anounces them, actual segment duration may be different due to
+   * approximations), with the exception of the last one (that usually is
+   * shorter).
+   * @returns {number}
+   */
+  getTargetSegmentDuration():
+    | {
+        /** Approximate duration of any segments but the last one in seconds. */
+        duration: number;
+        /**
+         * If `true`, the given duration should be relatively precize for all
+         * segments but the last one.
+         *
+         * If `false`, `duration` indicates only a general idea of what can be
+         * expected.
+         */
+        isPrecize: boolean;
+      }
+    | undefined;
+
   /**
    * Add segments to a RepresentationIndex that were predicted after parsing the
    * segment linked to `currentSegment`.

src/manifest/classes/update_period_in_place.ts

@@ -75,6 +75,10 @@ export default function updatePeriodInPlace(
       oldThumbnailTrack.width = newThumbnailTrack.width;
       oldThumbnailTrack.horizontalTiles = newThumbnailTrack.horizontalTiles;
       oldThumbnailTrack.verticalTiles = newThumbnailTrack.verticalTiles;
+      oldThumbnailTrack.start = newThumbnailTrack.start;
+      oldThumbnailTrack.end = newThumbnailTrack.end;
+      oldThumbnailTrack.segmentDuration = newThumbnailTrack.segmentDuration;
+      oldThumbnailTrack.thumbnailsPerSegment = newThumbnailTrack.thumbnailsPerSegment;
       oldThumbnailTrack.cdnMetadata = newThumbnailTrack.cdnMetadata;
       if (updateType === MANIFEST_UPDATE_TYPE.Full) {
         oldThumbnailTrack.index._replace(newThumbnailTrack.index);
@@ -88,6 +92,10 @@ export default function updatePeriodInPlace(
         width: oldThumbnailTrack.width,
         horizontalTiles: oldThumbnailTrack.horizontalTiles,
         verticalTiles: oldThumbnailTrack.verticalTiles,
+        start: oldThumbnailTrack.start,
+        end: oldThumbnailTrack.end,
+        segmentDuration: oldThumbnailTrack.segmentDuration,
+        thumbnailsPerSegment: oldThumbnailTrack.thumbnailsPerSegment,
       });
     }
   }
@@ -105,6 +113,10 @@ export default function updatePeriodInPlace(
         width: t.width,
         horizontalTiles: t.horizontalTiles,
         verticalTiles: t.verticalTiles,
+        start: t.start,
+        end: t.end,
+        segmentDuration: t.segmentDuration,
+        thumbnailsPerSegment: t.thumbnailsPerSegment,
       })),
     );
     oldPeriod.thumbnailTracks.push(...newThumbnailTracks);

src/manifest/types.ts

@@ -286,6 +286,61 @@ export interface IThumbnailTrackMetadata {
    * images contained vertically in a whole loaded thumbnail resource.
    */
   verticalTiles: number;
+  /**
+   * Starting `position` the first thumbnail of this thumbnail track applies to,
+   * if known.
+   */
+  start: number | undefined;
+  /**
+   * Ending `position` the last thumbnail of this thumbnail track applies to,
+   * if known.
+   */
+  end: number | undefined;
+  /**
+   * Individual thumbnails may be technically part of "segments" containing
+   * multiple consecutive thumbnails each.
+   *
+   * `thumbnailsPerSegment` is the number of `thumbnails` each segments have.
+   *
+   * For example you could have stored on the server a segment which is a grid
+   * of 2 x 3 (2 horizontal rows and 3 vertical columns) thumbnails, which the
+   * RxPlayer will load at once then "cut" the right way when calling
+   * `renderThumbnail`. In that example, `thumbnailsPerSegment` would be set to
+   * `6` (2*3).
+   *
+   * Note that the last segment of a content may contain less thumbnails as
+   * anounced here depending on the duration of the content.
+   *
+   * You may want to rely on this information alongside `segmentDuration` to
+   * construct a list of available thumbnails and/or of available segments of
+   * thumbnails.
+   */
+  thumbnailsPerSegment: number | undefined;
+  /**
+   * When loaded, thumbnails are part of so-called "segments" which may contain
+   * either a single thumbnail or a grid of them (@see `thumbnailsPerSegment`).
+   *
+   * This `segmentDuration` property indicates a duration in seconds each
+   * segment applies to. You might then want to divide that value by
+   * `thumbnailsPerSegment` to get the duration each thumbnail applies to.
+   *
+   * Note that the last segment of a content may have a lower duration depending
+   * on the duration of the content.
+   *
+   * Set to `undefined` either the duration is unknown or if the duration
+   * depends from segment to segments.
+   *
+   * For example, with a `start` set to `10`, an `end` set to `21`, a
+   * `thumbnailsPerSegment` set to `2` and a `SegmentDuration` set to
+   * `4`, there should be 3 segments, each with 2 thumbnails:
+   * 1. A segment of 2 thumbnails for the seconds: 10-14
+   *    (The first thumbnail in that segment for 10-12, the second for 12-14)
+   * 2. A segment of 2 thumbnails for the seconds: 14-18
+   *    (The first thumbnail in that segment for 14-16, the second for 16-18)
+   * 3. A segment of 2 thumbnails for the seconds: 18-21 (the end)
+   *    (The first thumbnail in that segment for 18-20, the second for 20-21)
+   */
+  segmentDuration: number | undefined;
 }
 
 export interface ILoadedThumbnailData {

src/parsers/manifest/dash/common/indexes/base.ts

@@ -433,6 +433,29 @@ export default class BaseRepresentationIndex implements IRepresentationIndex {
     log.warn("Cannot add predicted segments to a `BaseRepresentationIndex`");
   }
 
+  /**
+   * Returns the `duration` of each segment in the context of its Manifest (i.e.
+   * as the Manifest anounces them, actual segment duration may be different due
+   * to approximations), in seconds.
+   *
+   * NOTE: we could here do a median or a mean but I chose to be lazy (and
+   * more performant) by returning the duration of the first element instead.
+   * As `isPrecize` is `false`, the rest of the code should be notified that
+   * this is only an approximation.
+   * @returns {number}
+   */
+  getTargetSegmentDuration(): { duration: number; isPrecize: boolean } | undefined {
+    const { timeline, timescale } = this._index;
+    const firstElementInTimeline = timeline[0];
+    if (firstElementInTimeline === undefined) {
+      return undefined;
+    }
+    return {
+      duration: firstElementInTimeline.duration / timescale,
+      isPrecize: false,
+    };
+  }
+
   /**
    * Replace in-place this `BaseRepresentationIndex` information by the
    * information from another one.