From 517f6a07e84a252d60976960d90e0206f7003a9c Mon Sep 17 00:00:00 2001 From: Bernard Aboba Date: Wed, 3 Jan 2024 22:20:47 -0800 Subject: [PATCH 1/4] Extend EncodedVideoChunkMetadata for Spatial Scalability Fixes #619 Rebase and update of PR #654 --- index.src.html | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-) diff --git a/index.src.html b/index.src.html index 7a704560..db3937e4 100644 --- a/index.src.html +++ b/index.src.html @@ -133,6 +133,10 @@ :: A grouping of {{EncodedVideoChunk}}s whose timestamp cadence produces a particular framerate. See {{VideoEncoderConfig/scalabilityMode}}. +: Spatial Layer +:: A grouping of {{EncodedVideoChunk}}s which produces a particular + resolution. See {{VideoEncoderConfig/scalabilityMode}}. + : Progressive Image :: An image that supports decoding to multiple levels of detail, with lower levels becoming available while the encoded data is not yet fully buffered. @@ -1652,11 +1656,20 @@ |svc|.{{SvcOutputMetadata/temporalLayerId}}. 4. Assign |svc| to |chunkMetadata|.{{EncodedVideoChunkMetadata/svc}}. - 8. If |encoderConfig|.{{VideoEncoderConfig/alpha}} is set to `"keep"`: + 8. If |encoderConfig|.{{VideoEncoderConfig/scalabilityMode}} + describes multiple [=spatial layers=]: + 1. Let |svc| be a new {{SvcOutputMetadata}} instance. + 2. Let |spatial_layer_id| be the zero-based index describing the + spatial layer for |output|. + 3. Assign |spatial_layer_id| to + |svc|.{{SvcOutputMetadata/spatialLayerId}}. + 4. Assign |svc| to + |chunkMetadata|.{{EncodedVideoChunkMetadata/svc}}. + 9. If |encoderConfig|.{{VideoEncoderConfig/alpha}} is set to `"keep"`: 1. Let |alphaSideData| be the encoded alpha data in |output|. 2. Assign |alphaSideData| to |chunkMetadata|.{{EncodedVideoChunkMetadata/alphaSideData}}. - 9. Invoke {{VideoEncoder/[[output callback]]}} with |chunk| and + 10. Invoke {{VideoEncoder/[[output callback]]}} with |chunk| and |chunkMetadata|.
Reset VideoEncoder (with |exception|)
@@ -1704,6 +1717,9 @@ dictionary SvcOutputMetadata { unsigned long temporalLayerId; + unsigned long spatialLayerId; + unsigned long long frameId; + sequence dependencies; }; @@ -1723,7 +1739,17 @@ :: A number that identifies the [=temporal layer=] for the associated {{EncodedVideoChunk}}. +: spatialLayerId +:: A number that identifies the [=spatial layer=] for the associated + {{EncodedVideoChunk}}. + +: frameId +:: A number that identifies the associated {{EncodedVideoChunk}}. + +: dependencies +:: A sequence containing the {{frameId}} values that the associated {{EncodedVideoChunk}} depends on. + Configurations{#configurations} =============================== From c7dcedc761996d116f7a20900f277b0911a13dda Mon Sep 17 00:00:00 2001 From: Bernard Aboba Date: Tue, 9 Jan 2024 14:13:11 -0800 Subject: [PATCH 2/4] Move frameId and dependencies --- index.src.html | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/index.src.html b/index.src.html index db3937e4..049bf936 100644 --- a/index.src.html +++ b/index.src.html @@ -1713,13 +1713,13 @@ VideoDecoderConfig decoderConfig; SvcOutputMetadata svc; BufferSource alphaSideData; + unsigned long long frameId; + sequence dependencies; }; dictionary SvcOutputMetadata { unsigned long temporalLayerId; unsigned long spatialLayerId; - unsigned long long frameId; - sequence dependencies; }; From 16ba584e92284a6d70231241dfc944f7e933ba07 Mon Sep 17 00:00:00 2001 From: Bernard Aboba Date: Tue, 9 Jan 2024 14:22:56 -0800 Subject: [PATCH 3/4] Fix temporal/spatial layer assignments --- index.src.html | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/index.src.html b/index.src.html index 049bf936..36992194 100644 --- a/index.src.html +++ b/index.src.html @@ -1648,28 +1648,29 @@ 2. Assign |outputConfig| to {{VideoEncoder/[[active output config]]}}. 7. If |encoderConfig|.{{VideoEncoderConfig/scalabilityMode}} + describes multiple [=temporal layers=] or [=spatial layers=]: + 1. Let |svc| be a new {{SvcOutputMetadata}} instance. + 8. If |encoderConfig|.{{VideoEncoderConfig/scalabilityMode}} describes multiple [=temporal layers=]: - 1. Let |svc| be a new {{SvcOutputMetadata}} instance. - 2. Let |temporal_layer_id| be the zero-based index describing the + 1. Let |temporal_layer_id| be the zero-based index describing the temporal layer for |output|. - 3. Assign |temporal_layer_id| to + 2. Assign |temporal_layer_id| to |svc|.{{SvcOutputMetadata/temporalLayerId}}. - 4. Assign |svc| to - |chunkMetadata|.{{EncodedVideoChunkMetadata/svc}}. - 8. If |encoderConfig|.{{VideoEncoderConfig/scalabilityMode}} + 9. If |encoderConfig|.{{VideoEncoderConfig/scalabilityMode}} describes multiple [=spatial layers=]: - 1. Let |svc| be a new {{SvcOutputMetadata}} instance. - 2. Let |spatial_layer_id| be the zero-based index describing the + 1. Let |spatial_layer_id| be the zero-based index describing the spatial layer for |output|. - 3. Assign |spatial_layer_id| to + 2. Assign |spatial_layer_id| to |svc|.{{SvcOutputMetadata/spatialLayerId}}. - 4. Assign |svc| to + 10. If |encoderConfig|.{{VideoEncoderConfig/scalabilityMode}} + describes multiple [=temporal layers=] or [=spatial layers=]: + 1. Assign |svc| to |chunkMetadata|.{{EncodedVideoChunkMetadata/svc}}. - 9. If |encoderConfig|.{{VideoEncoderConfig/alpha}} is set to `"keep"`: + 11. If |encoderConfig|.{{VideoEncoderConfig/alpha}} is set to `"keep"`: 1. Let |alphaSideData| be the encoded alpha data in |output|. 2. Assign |alphaSideData| to |chunkMetadata|.{{EncodedVideoChunkMetadata/alphaSideData}}. - 10. Invoke {{VideoEncoder/[[output callback]]}} with |chunk| and + 12. Invoke {{VideoEncoder/[[output callback]]}} with |chunk| and |chunkMetadata|.
Reset VideoEncoder (with |exception|)
From 9efde67f1de0fdf0194f96bc8cb8f1eeb9197d80 Mon Sep 17 00:00:00 2001 From: Bernard Aboba Date: Tue, 9 Jan 2024 14:27:05 -0800 Subject: [PATCH 4/4] Move frameId and dependencies --- index.src.html | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/index.src.html b/index.src.html index 36992194..21638a6f 100644 --- a/index.src.html +++ b/index.src.html @@ -1736,6 +1736,12 @@ :: A {{BufferSource}} that contains the {{EncodedVideoChunk}}'s extra alpha channel data. +: frameId +:: A number that identifies the associated {{EncodedVideoChunk}}. + +: dependencies +:: A sequence containing the {{frameId}} values that the associated {{EncodedVideoChunk}} depends on. + : temporalLayerId :: A number that identifies the [=temporal layer=] for the associated {{EncodedVideoChunk}}. @@ -1743,13 +1749,6 @@ : spatialLayerId :: A number that identifies the [=spatial layer=] for the associated {{EncodedVideoChunk}}. - -: frameId -:: A number that identifies the associated {{EncodedVideoChunk}}. - - -: dependencies -:: A sequence containing the {{frameId}} values that the associated {{EncodedVideoChunk}} depends on. Configurations{#configurations} ===============================