From 1acc485ffba3c12886e146f03557068d898a1f48 Mon Sep 17 00:00:00 2001 From: Markus Handell Date: Fri, 20 Sep 2024 17:19:43 +0200 Subject: [PATCH 1/5] add WebRTC-specific interactions with capture/receive/RTP timestamps --- webrtc-extensions.js | 94 ++++++++++++++++++++++---------------------- 1 file changed, 48 insertions(+), 46 deletions(-) diff --git a/webrtc-extensions.js b/webrtc-extensions.js index 7cc3dc4..468cac8 100644 --- a/webrtc-extensions.js +++ b/webrtc-extensions.js @@ -1,50 +1,52 @@ var respecConfig = { - "group": "webrtc", - github: { - repoURL: "https://github.com/w3c/webrtc-extensions/", - branch: "main" - }, + "group": "webrtc", + github: { + repoURL: "https://github.com/w3c/webrtc-extensions/", + branch: "main" + }, latestVersion: null, - "xref": ["html", "webidl", "webrtc", "hr-time", "mediacapture-streams", "webrtc-stats", "infra", "dom"], - "specStatus": "ED", - editors: [ - // { name: "Your Name", url: "http://example.org/", - // company: "Your Company", companyURL: "http://example.com/" }, - { name: "Bernard Aboba", company: "Microsoft Corporation", - w3cid: "65611" - } - ], - formerEditors: [ - { name: "Henrik Boström", company: "Google", w3cid: "96936", retiredDate: "2021-02-01" } - ], - authors: [ - ], - wgPublicList: "public-webrtc", - otherLinks: [ - { - key: "Participate", - data: [ - { - value: "Mailing list", - href: "https://lists.w3.org/Archives/Public/public-webrtc/" - } - ] - } - ], - localBiblio: { - "IANA-STUN-6": { - "title": "STUN Error Codes", - "href": "https://www.iana.org/assignments/stun-parameters/stun-parameters.xhtml#stun-parameters-6", - "publisher": "IANA" - }, - "CRYPTEX": { - "aliasOf": "RFC9335" - }, - "RTP-EXT-CAPTURE-TIME": { - "title": "RTP Header Extension for Absolute Capture Time", - "publisher": "WebRTC Project", - href: "https://webrtc.googlesource.com/src/+/refs/heads/main/docs/native-code/rtp-hdrext/abs-capture-time", - "status": "Experimental RTP Header Extension" - } + "xref": ["html", "webidl", "webrtc", "hr-time", "mediacapture-streams", "webrtc-stats", "infra", "dom"], + "specStatus": "ED", + editors: [ + // { name: "Your Name", url: "http://example.org/", + // company: "Your Company", companyURL: "http://example.com/" }, + { + name: "Bernard Aboba", company: "Microsoft Corporation", + w3cid: "65611" + } + ], + formerEditors: [ + { name: "Henrik Boström", company: "Google", w3cid: "96936", retiredDate: "2021-02-01" } + ], + authors: [ + ], + wgPublicList: "public-webrtc", + otherLinks: [ + { + key: "Participate", + data: [ + { + value: "Mailing list", + href: "https://lists.w3.org/Archives/Public/public-webrtc/" + } + ] + } + ], + localBiblio: { + "IANA-STUN-6": { + "title": "STUN Error Codes", + "href": "https://www.iana.org/assignments/stun-parameters/stun-parameters.xhtml#stun-parameters-6", + "publisher": "IANA" + }, + "CRYPTEX": { + "aliasOf": "RFC9335" + }, + "RTP-EXT-CAPTURE-TIME": { + "title": "RTP Header Extension for Absolute Capture Time", + "publisher": "WebRTC Project", + href: "https://webrtc.googlesource.com/src/+/refs/heads/main/docs/native-code/rtp-hdrext/abs-capture-time", + "status": "Experimental RTP Header Extension" } + }, + "xref": ["webrtc", "hr-time", "mediacapture-streams", "webrtc-stats", "mediacapture-extensions"] } From 3ee4f434e6643a3f0a11ba2307fbcf91d81d930b Mon Sep 17 00:00:00 2001 From: Markus Handell Date: Wed, 18 Dec 2024 15:41:35 +0100 Subject: [PATCH 2/5] Rebase and update --- index.html | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/index.html b/index.html index 505067d..141d401 100644 --- a/index.html +++ b/index.html @@ -72,6 +72,19 @@

Terminology

The process of freeing a candidate is defined in [[RFC8445]] Section 8.3.

+ +

+ + The following terms are defined in mediacapture-extensions + video timestamp concepts +

+

@@ -1540,6 +1553,46 @@

Event summary

+
+

Timestamp behavior

+

RTCRtpSender timestamp effects on packet NTP and RTP timestamps

+

+ The user agent defines a frame timestamp being the same as the [=capture timestamp=] + of the frame being produced on the {{RTCRtpSender}} {{MediaStreamTrack}} if it is set. + If it is unset the user agent estimates a timestamp from the sent frame's [= presentation timestamp =] together with the + time it was received by the {{RTCRtpSender}}. +

+

+ The NTP and RTP timestamps of the encoded frame corresponding to the frame being produced on the {{RTCRtpSender}} + {{MediaStreamTrack}} are sourced from the [= frame timestamp =]. +

+

+ The [=RTP timestamp=] of frames being produced on the {{RTCRtpSender}} is ignored. +

+

RTCRtpReceiver timestamps

+

Remote capture timestamp

+

+ + For a frame produced in a {{RTCRtpReceiver}} track, the user agent computes a + remote capture timestamp. It is a best-effort estimate of the local capture + time on the sender translated to the receiver clock, and can use methods like using RTCP SR + as specified in [[?RFC3550]] Section 6.4.1, or by other alternative means if use by RTCP SR + isn't feasible. +

+ Each frame's [= capture timestamp =] is set to the [= remote capture timestamp =], if available. +

+

Received RTP timestamp

+

+ For a frame produced in a {{RTCRtpReceiver}} track, the frame's [=RTP timestamp=] is set + from the RTP timestamp of constituent packets of the corresponding received encoded frame. +

+

Receive timestamp

+

+ For a frame produced in a {{RTCRtpReceiver}} track, the [=receive timestamp=] is set + as the time the corresponding encoded frame was received by the platform, i.e. the time at which the + last packet belonging to this frame was received over the network. +

+

Security Considerations From 92a79db41a4d39cb310fa49f0d371a419d9aaa8a Mon Sep 17 00:00:00 2001 From: Markus Handell Date: Wed, 18 Dec 2024 16:23:18 +0100 Subject: [PATCH 3/5] Fix some style stuff & undo webrtc-extensions.js changes. --- index.html | 6 +-- webrtc-extensions.js | 94 ++++++++++++++++++++++---------------------- 2 files changed, 49 insertions(+), 51 deletions(-) diff --git a/index.html b/index.html index 141d401..2268dc8 100644 --- a/index.html +++ b/index.html @@ -1559,12 +1559,12 @@

RTCRtpSender timestamp effects on packet NTP and RTP timestamps

The user agent defines a frame timestamp being the same as the [=capture timestamp=] of the frame being produced on the {{RTCRtpSender}} {{MediaStreamTrack}} if it is set. - If it is unset the user agent estimates a timestamp from the sent frame's [= presentation timestamp =] together with the + If it is unset the user agent estimates a timestamp from the sent frame's [=presentation timestamp=] together with the time it was received by the {{RTCRtpSender}}.

The NTP and RTP timestamps of the encoded frame corresponding to the frame being produced on the {{RTCRtpSender}} - {{MediaStreamTrack}} are sourced from the [= frame timestamp =]. + {{MediaStreamTrack}} are sourced from the [=frame timestamp=].

The [=RTP timestamp=] of frames being produced on the {{RTCRtpSender}} is ignored. @@ -1579,7 +1579,7 @@

Remote capture timestamp

as specified in [[?RFC3550]] Section 6.4.1, or by other alternative means if use by RTCP SR isn't feasible.

- Each frame's [= capture timestamp =] is set to the [= remote capture timestamp =], if available. + Each frame's [=capture timestamp=] is set to the [=remote capture timestamp=], if available.

Received RTP timestamp

diff --git a/webrtc-extensions.js b/webrtc-extensions.js index 468cac8..7cc3dc4 100644 --- a/webrtc-extensions.js +++ b/webrtc-extensions.js @@ -1,52 +1,50 @@ var respecConfig = { - "group": "webrtc", - github: { - repoURL: "https://github.com/w3c/webrtc-extensions/", - branch: "main" - }, - latestVersion: null, - "xref": ["html", "webidl", "webrtc", "hr-time", "mediacapture-streams", "webrtc-stats", "infra", "dom"], - "specStatus": "ED", - editors: [ - // { name: "Your Name", url: "http://example.org/", - // company: "Your Company", companyURL: "http://example.com/" }, - { - name: "Bernard Aboba", company: "Microsoft Corporation", - w3cid: "65611" - } - ], - formerEditors: [ - { name: "Henrik Boström", company: "Google", w3cid: "96936", retiredDate: "2021-02-01" } - ], - authors: [ - ], - wgPublicList: "public-webrtc", - otherLinks: [ - { - key: "Participate", - data: [ - { - value: "Mailing list", - href: "https://lists.w3.org/Archives/Public/public-webrtc/" - } - ] - } - ], - localBiblio: { - "IANA-STUN-6": { - "title": "STUN Error Codes", - "href": "https://www.iana.org/assignments/stun-parameters/stun-parameters.xhtml#stun-parameters-6", - "publisher": "IANA" + "group": "webrtc", + github: { + repoURL: "https://github.com/w3c/webrtc-extensions/", + branch: "main" }, - "CRYPTEX": { - "aliasOf": "RFC9335" - }, - "RTP-EXT-CAPTURE-TIME": { - "title": "RTP Header Extension for Absolute Capture Time", - "publisher": "WebRTC Project", - href: "https://webrtc.googlesource.com/src/+/refs/heads/main/docs/native-code/rtp-hdrext/abs-capture-time", - "status": "Experimental RTP Header Extension" + latestVersion: null, + "xref": ["html", "webidl", "webrtc", "hr-time", "mediacapture-streams", "webrtc-stats", "infra", "dom"], + "specStatus": "ED", + editors: [ + // { name: "Your Name", url: "http://example.org/", + // company: "Your Company", companyURL: "http://example.com/" }, + { name: "Bernard Aboba", company: "Microsoft Corporation", + w3cid: "65611" + } + ], + formerEditors: [ + { name: "Henrik Boström", company: "Google", w3cid: "96936", retiredDate: "2021-02-01" } + ], + authors: [ + ], + wgPublicList: "public-webrtc", + otherLinks: [ + { + key: "Participate", + data: [ + { + value: "Mailing list", + href: "https://lists.w3.org/Archives/Public/public-webrtc/" + } + ] + } + ], + localBiblio: { + "IANA-STUN-6": { + "title": "STUN Error Codes", + "href": "https://www.iana.org/assignments/stun-parameters/stun-parameters.xhtml#stun-parameters-6", + "publisher": "IANA" + }, + "CRYPTEX": { + "aliasOf": "RFC9335" + }, + "RTP-EXT-CAPTURE-TIME": { + "title": "RTP Header Extension for Absolute Capture Time", + "publisher": "WebRTC Project", + href: "https://webrtc.googlesource.com/src/+/refs/heads/main/docs/native-code/rtp-hdrext/abs-capture-time", + "status": "Experimental RTP Header Extension" + } } - }, - "xref": ["webrtc", "hr-time", "mediacapture-streams", "webrtc-stats", "mediacapture-extensions"] } From 18a0c80346628c64340e3447d116ef8f462ae129 Mon Sep 17 00:00:00 2001 From: Markus Handell Date: Fri, 10 Jan 2025 14:48:35 +0100 Subject: [PATCH 4/5] Address reviewer comments. Use SHOULD/MUST instead of is. Remove informative as things should be observable. --- index.html | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/index.html b/index.html index 2268dc8..aaafdc9 100644 --- a/index.html +++ b/index.html @@ -1553,21 +1553,21 @@

Event summary

-
+

Timestamp behavior

-

RTCRtpSender timestamp effects on packet NTP and RTP timestamps

+

RTCRtpSender video track frame timestamp effects on outgoing packet NTP and RTP timestamps

The user agent defines a frame timestamp being the same as the [=capture timestamp=] of the frame being produced on the {{RTCRtpSender}} {{MediaStreamTrack}} if it is set. - If it is unset the user agent estimates a timestamp from the sent frame's [=presentation timestamp=] together with the + If it is unset the user agent MUST estimate a timestamp from the sent frame's [=presentation timestamp=] together with the time it was received by the {{RTCRtpSender}}.

The NTP and RTP timestamps of the encoded frame corresponding to the frame being produced on the {{RTCRtpSender}} - {{MediaStreamTrack}} are sourced from the [=frame timestamp=]. + {{MediaStreamTrack}} MUST be sourced from the [=frame timestamp=].

- The [=RTP timestamp=] of frames being produced on the {{RTCRtpSender}} is ignored. + The [=RTP timestamp=] of frames produced on the {{RTCRtpSender}} {{MediaStreamTrack}} SHOULD not be used by the {{RTCRtpSender}}.

RTCRtpReceiver timestamps

Remote capture timestamp

@@ -1583,12 +1583,12 @@

Remote capture timestamp

Received RTP timestamp

- For a frame produced in a {{RTCRtpReceiver}} track, the frame's [=RTP timestamp=] is set + For a video frame produced in a {{RTCRtpReceiver}} track, the frame's [=RTP timestamp=] MUST be set from the RTP timestamp of constituent packets of the corresponding received encoded frame.

Receive timestamp

- For a frame produced in a {{RTCRtpReceiver}} track, the [=receive timestamp=] is set + For a frame produced in a {{RTCRtpReceiver}} track, the [=receive timestamp=] MUST be set as the time the corresponding encoded frame was received by the platform, i.e. the time at which the last packet belonging to this frame was received over the network.

From 427dec6cb78a66276e7858581841ce167af1953e Mon Sep 17 00:00:00 2001 From: Markus Handell Date: Fri, 10 Jan 2025 15:23:33 +0100 Subject: [PATCH 5/5] Offline review with henbos@: removed ambiguous text. --- index.html | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/index.html b/index.html index aaafdc9..d26e4d1 100644 --- a/index.html +++ b/index.html @@ -1560,15 +1560,12 @@

RTCRtpSender video track frame timestamp effects on outgoing packet NTP and The user agent defines a frame timestamp being the same as the [=capture timestamp=] of the frame being produced on the {{RTCRtpSender}} {{MediaStreamTrack}} if it is set. If it is unset the user agent MUST estimate a timestamp from the sent frame's [=presentation timestamp=] together with the - time it was received by the {{RTCRtpSender}}. + time it was received by the {{RTCRtpSender}}. This estimate MUST NOT be based on [= RTP timestamp =].

The NTP and RTP timestamps of the encoded frame corresponding to the frame being produced on the {{RTCRtpSender}} {{MediaStreamTrack}} MUST be sourced from the [=frame timestamp=].

-

- The [=RTP timestamp=] of frames produced on the {{RTCRtpSender}} {{MediaStreamTrack}} SHOULD not be used by the {{RTCRtpSender}}. -

RTCRtpReceiver timestamps

Remote capture timestamp