Feat: add an option to play without audio or video if audio or video failed #1624

Florent-Bouisset · 2025-01-06T09:28:11Z

Currently, the player throws an error if either all audio or all video tracks are not playable. This PR introduces two new options, onAudioTrackNotPlayable and onVideoTrackNotPlayable, to the loadVideo method. These options allow the player to continue playback even when all audio or video tracks are not playable. In such cases, the content will play without audio or video, respectively, instead of failing.

API

loadVideo(options) options object has two more keys:

[optional]onAudioTrackNotPlayable : "continue" or "error". Default if not set: "continue"
[optional]onVideoTrackNotPlayable : "continue" or "error". Default if not set: "continue"

Usage

  player.loadVideo({
    transport: "dash",
    url: "https://dash.akamaized.net/dash264/TestCasesDolby/1/Living_Room_1080p_20_96k_25fps.mpd",
    autoPlay: true,
    onAudioTrackNotPlayable: "continue",
    onVideoTrackNotPlayable: "error",
  });

Example

https://codesandbox.io/p/sandbox/amazing-pare-kdf65f

Florent-Bouisset · 2025-01-06T16:12:53Z

relates to #1602

peaBerberian · 2025-01-06T17:12:20Z

onAudioTrackNotPlayable and onVideoTrackNotPlayable

Shouldn't it be "tracks" here?

Default if not set: "continue"

Doesn't it break the API this way? Its complex but it changes the default behavior on which some applications may rely? Wouldn't it be safer to default it to "error"?

peaBerberian · 2025-01-06T17:29:45Z

doc/api/Loading_a_Content.md

+
+_defaults_: `"continue"`
+
+Specifies the behavior when all audio tracks are not playable.


We could clarify by giving some example cases below

What kind of example could be relevant here? Doesn't the description of the option provides clear indication on what the player behavior is?

"continue": The player will proceed to play the content without audio.

"error": The player will throw an error to indicate that the audio tracks could not be
played.

Or maybe you want me to add an example that describes how all tracks can be unplayable (codec not supported, license not containing the keys for audio/video, ...)

Or maybe you want me to add an example that describes how all tracks can be unplayable (codec not supported, license not containing the keys for audio/video, ...)

Yes, this

Florent-Bouisset · 2025-01-07T13:14:29Z

onAudioTrackNotPlayable

Yes I think I should default it to "error" instead.

Florent-Bouisset · 2025-01-09T10:22:34Z

onAudioTrackNotPlayable and onVideoTrackNotPlayable

Shouldn't it be "tracks" here?

Default if not set: "continue"

Doesn't it break the API this way? Its complex but it changes the default behavior on which some applications may rely? Wouldn't it be safer to default it to "error"?

I have updated it.

peaBerberian · 2025-01-09T11:07:14Z

src/core/stream/period/period_stream.ts

+    return representations[0].getMimeTypeString();
+  } else if (adaptation.representations.length > 0) {
+    return adaptation.representations[0].getMimeTypeString();
+  } else {


Just to be sure, this one else should never be able to be entered no?

peaBerberian · 2025-01-09T11:10:47Z

src/main_thread/tracks_store/tracks_store.ts

@@ -1469,6 +1486,25 @@ function toExposedPeriod(p: IPeriodMetadata): IPeriod {
  return { start: p.start, end: p.end, id: p.id };
 }

+function findNextPlayableAdaptation(


From that function's point of view, I'm under the impression that this is not really the "next" playable Adaptation, just the first playable adaptation, no?

peaBerberian · 2025-01-09T11:11:46Z

src/main_thread/tracks_store/tracks_store.ts

+        nextAdaptation === undefined &&
+        bufferType === "audio" &&
+        this.onAudioTracksNotPlayable === "continue" &&
+        findNextPlayableAdaptation(period, "video") !== undefined


Why are we doing this call? We just have to check that the current video track is defined no?

Also what happens if we were in audio-only mode here?

peaBerberian · 2025-01-09T11:12:30Z

src/main_thread/tracks_store/tracks_store.ts

+        nextAdaptation === undefined &&
+        bufferType === "video" &&
+        this.onVideoTracksNotPlayable === "continue" &&
+        findNextPlayableAdaptation(period, "audio") !== undefined


Same, we should just check that there's currently an audio track no?

peaBerberian · 2025-01-09T11:20:22Z

As this adds a similar logic in the TrackStore + Manifest + Init, and as the manifest is more or less intended to just regroup metadata, I'm wondering if it would not be easier to follow if that option was mainly handled by a part of the Init (and TrackStore I guess for its own thing) : Like the Manifest would parse the content regardless, and the Init would investigate the issues seen while parsing and decide whether to error or warn and continue based on its options.

What do you think?

github-actions · 2025-01-29T16:21:54Z

Automated performance checks have been performed on commit 168ac23a2c6e5dc5bae395c9d92809c9a52ed54c with the base branch dev.

Tests results

✅ Tests have passed.

Performance tests 1st run output

No significative change in performance for tests:

Name	Mean	Median
loading	19.31ms -> 19.52ms (-0.204ms, z: 0.26401)	26.55ms -> 26.70ms
seeking	14.01ms -> 11.30ms (2.710ms, z: 0.18678)	11.25ms -> 11.25ms
audio-track-reload	25.71ms -> 25.73ms (-0.022ms, z: 0.00379)	37.65ms -> 37.65ms
cold loading multithread	47.55ms -> 46.71ms (0.837ms, z: 10.77750)	69.60ms -> 68.40ms
seeking multithread	15.95ms -> 11.95ms (3.997ms, z: 0.80564)	10.50ms -> 10.50ms
audio-track-reload multithread	25.74ms -> 25.71ms (0.024ms, z: 1.49764)	37.95ms -> 37.80ms
hot loading multithread	15.04ms -> 15.02ms (0.013ms, z: 2.03549)	21.90ms -> 21.75ms

If you want to skip performance checks for latter commits, add the skip-performance-checks label to this Pull Request.

…o is not supported

…tPlayable to "error"

github-actions · 2025-01-29T17:23:51Z

Automated performance checks have been performed on commit 5eb15f4f29a23ec1b8774929f8e37d8b0c51b6d2 with the base branch dev.

Tests results

✅ Tests have passed.

Performance tests 1st run output

No significative change in performance for tests:

Name	Mean	Median
loading	19.42ms -> 19.50ms (-0.075ms, z: 1.47093)	26.85ms -> 26.70ms
seeking	16.68ms -> 17.28ms (-0.603ms, z: 0.25378)	11.25ms -> 11.25ms
audio-track-reload	25.67ms -> 25.76ms (-0.088ms, z: 0.08821)	37.65ms -> 37.65ms
cold loading multithread	47.63ms -> 46.98ms (0.658ms, z: 10.77232)	69.90ms -> 68.70ms
seeking multithread	15.95ms -> 19.94ms (-3.991ms, z: 1.73102)	10.35ms -> 10.35ms
audio-track-reload multithread	25.92ms -> 25.54ms (0.375ms, z: 2.54631)	37.80ms -> 37.65ms
hot loading multithread	15.08ms -> 14.95ms (0.130ms, z: 3.13205)	21.90ms -> 21.60ms

If you want to skip performance checks for latter commits, add the skip-performance-checks label to this Pull Request.

…demo

…dia has no track.

github-actions · 2025-01-31T10:04:07Z

Automated performance checks have been performed on commit dc7bfe735dfd50886b0db46aeb0e6a67ad1ac647 with the base branch dev.

Tests results

✅ Tests have passed.

Performance tests 1st run output

No significative change in performance for tests:

Name	Mean	Median
loading	19.29ms -> 19.51ms (-0.214ms, z: 3.58239)	26.40ms -> 26.85ms
seeking	14.60ms -> 14.67ms (-0.061ms, z: 2.14748)	11.25ms -> 11.40ms
audio-track-reload	25.86ms -> 25.84ms (0.027ms, z: 0.81078)	37.80ms -> 37.80ms
cold loading multithread	47.66ms -> 46.89ms (0.764ms, z: 9.64433)	69.75ms -> 68.55ms
seeking multithread	17.31ms -> 18.63ms (-1.324ms, z: 0.16955)	10.35ms -> 10.35ms
audio-track-reload multithread	25.87ms -> 25.75ms (0.118ms, z: 1.94454)	38.10ms -> 37.80ms
hot loading multithread	15.17ms -> 15.06ms (0.111ms, z: 1.80062)	22.05ms -> 21.90ms

If you want to skip performance checks for latter commits, add the skip-performance-checks label to this Pull Request.

peaBerberian · 2025-02-06T15:56:46Z

doc/api/Player_Events.md

+
+_payload type_: `Object`
+
+Emitted when no tracks of a particular type can be selected for a period.


We might want to indicate that this happen when the corresponding options are set, and in which condition it arises

Else a developer won't understand what this event is for and whether it should handle it

peaBerberian · 2025-02-06T15:58:53Z

src/main_thread/api/public_api.ts

      this._priv_onFatalError(err, contentInfos);
    });

-    contentInfos.tracksStore.onManifestUpdate(manifest);
+    tracksStore.addEventListener("noPlayableTrack", (trackType) => {


The payload name is not right

peaBerberian · 2025-02-06T16:04:39Z

src/manifest/classes/period.ts

@@ -135,6 +98,30 @@ export default class Period implements IPeriodMetadata {
    this.streamEvents = args.streamEvents === undefined ? [] : args.streamEvents;
  }

+  createAdaptationsObject(


Is the intention to create a private method that is only called at the start of the Period's instantiation?

If it is, I have some issues:

it's not marked as a private method

I tend to think that it is a bad idea to call a method before the instantiation was succesfully done because we're in a partial state in which even TypeScript might not see that the current instance is not fully constructed / initialized. It's a difficult state to think about.

As we're not even relying on the Period's context it seems here, can't we just make that a regular function?

peaBerberian · 2025-02-06T16:06:21Z

src/manifest/classes/period.ts

    }, {});
+    // this.checkIfStreamIsSupported(hasSupportedMedia);


peaBerberian · 2025-02-06T16:09:59Z

src/main_thread/tracks_store/tracks_store.ts

@@ -158,6 +171,49 @@ export default class TracksStore extends EventEmitter<ITracksStoreEvents> {
    /** Periods which have just been added. */
    const addedPeriods: ITSPeriodObject[] = [];

+    for (const period of periods) {
+      ["audio" as const, "video" as const].forEach((ttype: ITrackType) => {


Maybe we could just isolate a good chunk of it to its own function

peaBerberian · 2025-02-06T16:11:28Z

src/main_thread/tracks_store/tracks_store.ts

+          if (
+            firstPlayableAdaptation === undefined &&
+            ttype === "audio" &&
+            this.onAudioTracksNotPlayable === "continue"


Might have been easier to follow as something like this.onTracksNotPlayable[ttype] === continue? (suggestion)

peaBerberian · 2025-02-06T16:12:09Z

src/main_thread/tracks_store/tracks_store.ts

+            // Video is not playable but audio may be playable, let's continue the playback.
+            this.trigger("warning", err);
+          } else if (firstPlayableAdaptation !== undefined) {
+            this.trigger("warning", err);


We cannot make TypeScript understand that this does not happen?

peaBerberian · 2025-02-06T16:13:09Z

src/main_thread/tracks_store/tracks_store.ts

+          } else if (firstPlayableAdaptation !== undefined) {
+            this.trigger("warning", err);
+          } else {
+            this.trigger("error", err);


Shouldn't we also just abort the trackstore here? That's what seem to be done for NO_PLAYABLE_REPRESENTATION

peaBerberian · 2025-02-06T16:20:12Z

src/main_thread/tracks_store/tracks_store.ts

+          adaptation,
+          switchingMode: DEFAULT_VIDEO_TRACK_SWITCHING_MODE,
+          lockedRepresentations,
+        } as IVideoStoredSettings;


Why is the as needed here?

peaBerberian · 2025-02-06T16:21:00Z

src/main_thread/tracks_store/tracks_store.ts

+   * @param type
+   * @returns
+   */
+  private resetSelectedTrackIfNotAvailableAnymore(


peaBerberian · 2025-02-06T16:22:50Z

src/main_thread/tracks_store/tracks_store.ts

@@ -481,17 +498,119 @@ export default class TracksStore extends EventEmitter<ITracksStoreEvents> {
      if (
        periodObj.isPeriodAdvertised &&
        trackObj.dispatcher !== null &&
-        !trackObj.dispatcher.hasSetTrack() &&
-        trackObj.storedSettings !== undefined


Wait why are we removing this? I guess (though I don't remember why) it was placed on purpose as a guard and it has different semantics than null

peaBerberian · 2025-02-06T16:24:16Z

src/main_thread/tracks_store/tracks_store.ts

      ) {
-        trackObj.dispatcher.updateTrack(trackObj.storedSettings);
+        trackObj.dispatcher.updateTrack(trackObj.storedSettings ?? null);


I'm not sure I get it here.

To me, if you don't want a type of media, storedSettings should always be set explicitely to null. If storedSettings is ever undefined, it should always mean that we don't know yet

peaBerberian · 2025-02-06T16:25:26Z

src/main_thread/tracks_store/tracks_store.ts

+  noPlayableTrack: INoPlayableTrack;
+}
+
+interface INoPlayableTrack {


With an EventPayload suffix maybe?

peaBerberian · 2025-02-06T16:29:04Z

src/main_thread/tracks_store/tracks_store.ts

+      period.adaptations[bufferType].length > 0;
+    const firstPlayableAdaptation = findFirstPlayableAdaptation(period, bufferType);
+
+    if (!periodHasAdaptationForType) {


Weird case we would be in, an "onNoPlayableRepresentation" call for a buffer type we don't have

peaBerberian · 2025-02-06T16:29:21Z

src/main_thread/tracks_store/tracks_store.ts

+    }
+
+    if (bufferType === "text") {
+      return;


peaBerberian · 2025-02-06T16:32:18Z

src/main_thread/tracks_store/tracks_store.ts

+    ) {
+      // Audio is not playable but video may be playable, let's continue the playback.
+      log.warn(`TS: No playable audio, continuing without audio`);
+      this.trigger("noPlayableTrack", {


/!\ this event might call the application's code synchronously and completely change the tracksStore's state:

it might be disposed

the track choices might have completely changed

I signal this because I've seen no guard against those possibilities, we just assume we're in the same state.

Florent-Bouisset added the API Relative to the RxPlayer's API label Jan 6, 2025

Florent-Bouisset force-pushed the feat/play-without-audio-or-video branch from 724b4a3 to 5a073f9 Compare January 6, 2025 09:30

peaBerberian reviewed Jan 6, 2025

View reviewed changes

Florent-Bouisset force-pushed the feat/play-without-audio-or-video branch from ef25c83 to 4b6acd7 Compare January 9, 2025 10:39

peaBerberian reviewed Jan 9, 2025

View reviewed changes

Florent-Bouisset added 12 commits January 29, 2025 18:10

handle unsupported codec

c95b763

do not throw directly a "no playable representation" if audio or vide…

8a4cea6

…o is not supported

update API to pass down props

77e62cb

add documentation

a1644be

add unit test and fix existing

09f48a9

update behavior for trackstore and multi threading

a01fd72

change option name

eb7314e

change default value for onVideoTracksNotPlayable and onAudioTracksNo…

867dea0

…tPlayable to "error"

move noPlayableTrack error in the trackStore

0b1e69d

remove unsupportedAdaptations from period class constructor

bd3801b

remove warnings concept from manifest

fd34903

decouple logic from the transport function

3a63261

Florent-Bouisset added 2 commits January 30, 2025 10:18

add the onAudioTracksNotPlayable and onVideoTracksNotPlayable to the …

ccbe9e5

…demo

fix case where setting a null video adaptation

923dac7

Florent-Bouisset force-pushed the feat/play-without-audio-or-video branch from 5eb15f4 to 5e29ebe Compare January 30, 2025 13:52

add an event "noPlayableTrack" to signal application that a currentMe…

dc7bfe7

…dia has no track.

Florent-Bouisset force-pushed the feat/play-without-audio-or-video branch from 5e29ebe to dc7bfe7 Compare January 31, 2025 09:37

peaBerberian reviewed Feb 6, 2025

View reviewed changes

src/manifest/classes/period.ts

}, {});

// this.checkIfStreamIsSupported(hasSupportedMedia);

Copy link

Collaborator

peaBerberian Feb 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gné?

peaBerberian reviewed Feb 6, 2025

View reviewed changes

src/main_thread/tracks_store/tracks_store.ts

}

if (bufferType === "text") {

return;

Copy link

Collaborator

peaBerberian Feb 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why

peaBerberian reviewed Feb 6, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat: add an option to play without audio or video if audio or video failed #1624

Feat: add an option to play without audio or video if audio or video failed #1624

Florent-Bouisset commented Jan 6, 2025

Florent-Bouisset commented Jan 6, 2025

peaBerberian commented Jan 6, 2025

peaBerberian Jan 6, 2025

Florent-Bouisset Jan 9, 2025

peaBerberian Jan 9, 2025

Florent-Bouisset commented Jan 7, 2025

Florent-Bouisset commented Jan 9, 2025

peaBerberian Jan 9, 2025

peaBerberian Jan 9, 2025

peaBerberian Jan 9, 2025

peaBerberian Jan 9, 2025

peaBerberian commented Jan 9, 2025 •

edited

Loading

github-actions bot commented Jan 29, 2025

github-actions bot commented Jan 29, 2025

github-actions bot commented Jan 31, 2025

peaBerberian Feb 6, 2025

peaBerberian Feb 6, 2025

peaBerberian Feb 6, 2025

peaBerberian Feb 6, 2025

peaBerberian Feb 6, 2025

peaBerberian Feb 6, 2025

peaBerberian Feb 6, 2025

peaBerberian Feb 6, 2025

peaBerberian Feb 6, 2025

peaBerberian Feb 6, 2025

peaBerberian Feb 6, 2025 •

edited

Loading

peaBerberian Feb 6, 2025

peaBerberian Feb 6, 2025

peaBerberian Feb 6, 2025

peaBerberian Feb 6, 2025

peaBerberian Feb 6, 2025 •

edited

Loading


		_defaults_: `"continue"`

		Specifies the behavior when all audio tracks are not playable.


		_payload type_: `Object`

		Emitted when no tracks of a particular type can be selected for a period.

Feat: add an option to play without audio or video if audio or video failed #1624

Are you sure you want to change the base?

Feat: add an option to play without audio or video if audio or video failed #1624

Conversation

Florent-Bouisset commented Jan 6, 2025

API

Usage

Example

Florent-Bouisset commented Jan 6, 2025

peaBerberian commented Jan 6, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Florent-Bouisset commented Jan 7, 2025

Florent-Bouisset commented Jan 9, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peaBerberian commented Jan 9, 2025 • edited Loading

github-actions bot commented Jan 29, 2025

Tests results

Performance tests 1st run output

github-actions bot commented Jan 29, 2025

Tests results

Performance tests 1st run output

github-actions bot commented Jan 31, 2025

Tests results

Performance tests 1st run output

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peaBerberian Feb 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peaBerberian Feb 6, 2025 • edited Loading

Choose a reason for hiding this comment

peaBerberian commented Jan 9, 2025 •

edited

Loading

peaBerberian Feb 6, 2025 •

edited

Loading

peaBerberian Feb 6, 2025 •

edited

Loading