From 17e6cf208fade02cd7fad1a7c4486b7eb3f06ca9 Mon Sep 17 00:00:00 2001 From: Youenn Fablet Date: Wed, 30 Apr 2025 12:12:24 +0200 Subject: [PATCH] Add SFrame packetization handling for SFrameTransform Introduce RTCRtpScriptTransformType with "sframe" value. --- index.bs | 76 +++++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 64 insertions(+), 12 deletions(-) diff --git a/index.bs b/index.bs index 7d7dd73..abc3524 100644 --- a/index.bs +++ b/index.bs @@ -59,6 +59,9 @@ spec:webidl; type:dfn; text:resolve 
 spec:streams; type:interface; text:ReadableStream
+
+url: https://w3c.github.io/webrtc-pc/#dfn-update-the-negotiation-needed-flag; text: update the negotiation-needed flag; type: dfn; spec: WEBRTC
+
# Introduction # {#introduction} @@ -138,6 +141,7 @@ The readEncodedData algorithm is given a |rtcObject| as p 1. Increment |rtcObject|.`[[lastEnqueuedFrameCounter]]` by 1. 1. Let |frame| be the newly produced frame. 1. Set |frame|.`[[owner]]` to |rtcObject|. +1. Set |this|.`[[writable]]` to |this|.`[[transform]]`.`[[writable]]`. 1. Set |frame|.`[[counter]]` to |rtcObject|.`[[lastEnqueuedFrameCounter]]`. 1. If the frame has been produced by a {{RTCRtpReceiver}}: 1. If the relevant RTP packet contains the @@ -150,6 +154,7 @@ The readEncodedData algorithm is given a |rtcObject| as p [[RTP-EXT-CAPTURE-TIME#timestamp-interpolation|timestamp interpolation]] and set |frame|.`[[senderCaptureTimeOffset]]` to the most recent value that was present. 1. Otherwise, set |frame|.`[[captureTime]]` to undefined and set |frame|.`[[senderCaptureTimeOffset]]` to undefined. + 1. If |frame| was produced by a [=SFrame depacketizer=], set |frame|.`[[useSFrame]]` to true. 1. If the frame has been produced by a {{RTCRtpSender}}, set |frame|.`[[captureTime]]` to the capture timestamp using the methodology described in [[RTP-EXT-CAPTURE-TIME#absolute-capture-timestamp]] and set frame.`[[senderCaptureTimeOffset]]` to undefined. @@ -162,8 +167,10 @@ The writeEncodedData algorithm is given a |rtcObject| as 1. Let |data| be |frame|.`[[data]]`. 1. Let |serializedFrame| be [$StructuredSerializeWithTransfer$](|frame|, « |data| »). 1. Let |frameCopy| be [$StructuredDeserializeWithTransfer$](|serializedFrame|, |frame|'s [=relevant realm=]). +1. If |frame|.`[[useSFrame]]` is true, set |frameCopy|.`[[useSFrame]]` to true. 1. Enqueue |frameCopy| for processing as if it came directly from the encoded data source, by running one of the following steps: * If |rtcObject| is a {{RTCRtpSender}}, enqueue |frameCopy| to |rtcObject|'s packetizer, to be processed [=in parallel=]. + If |frameCopy|.`[[useSFrame]]` is true, |rtcObject|'s MUST use a [=SFrame packetizer=] or skip processing of |frameCopy|. * If |rtcObject| is a {{RTCRtpReceiver}}, enqueue |frameCopy| it to |rtcObject|'s decoder, to be processed [=in parallel=]. 1. Return [=a promise resolved with=] undefined. @@ -181,6 +188,7 @@ A RTCRtpTransform has private slots: * `[[readable]]` of type {{ReadableStream}}. * `[[writable]]` of type {{WritableStream}}. * `[[owner]]` of type {{RTCRtpSender}} or {{RTCRtpReceiver}}, initialized to null. +* `[[useSFrame]]` of type boolean. // FIXME: Decide whether augmenting this boolean with either cipher suite support or whether doing frame vs. packet based encryption. Each RTCRtpTransform has an association steps set, which is empty by default. @@ -199,6 +207,16 @@ The `transform` setter steps are: 1. [=AbortSignal/Add=] the [$chain transform algorithm$] to [=this=].`[[pipeToController]]`'s [=AbortController/signal=]. 2. [=AbortController/signal abort=] on [=this=].`[[pipeToController]]`. 1. Else, run the [$chain transform algorithm$] steps. +1. If [=this=] is a {{RTCRtpSender}}, run the following substeps: + 1. Let |useSFrame| be true if [=this=] is configured to use a [=SFrame packetizer=] and false otherwise. + 1. If |useSFrame| is equal to |checkedTransform|.`[[useSFrame]]`, abort these substeps. + 1. Configure [=this=]'s packetizer to use SFrame if |checkedTransform|.`[[useSFrame]]` is true and to not use SFrame if |checkedTransform|.`[[useSFrame]]` is false. + 1. [=Update the negotiation-needed flag=] for [=this=]'s connection. +1. Otherwise, run the following steps: + 1. Let |useSFrame| be true if [=this=] is configured to use a [=SFrame depacketizer=] and false otherwise. + 1. If |useSFrame| is equal to |checkedTransform|.`[[useSFrame]]`, abort these substeps. + 1. Configure [=this=]'s depacketizer to use SFrame if |checkedTransform|.`[[useSFrame]]` is true and to not use SFrame if |checkedTransform|.`[[useSFrame]]` is false. + 1. [=Update the negotiation-needed flag=] for [=this=]'s connection. 1. Set [=this=].`[[pipeToController]]` to |newPipeToController|. 1. Set [=this=].`[[transform]]` to |transform|. 1. Run the steps in the set of [$association steps$] of |transform| with [=this=]. @@ -248,7 +266,8 @@ SFrameTransform includes GenericTransformStream; enum SFrameTransformErrorEventType { "authentication", "keyID", - "syntax" + "syntax", + "packetization" }; [Exposed=(Window,DedicatedWorker)] @@ -275,13 +294,19 @@ The n 5. Set |this|.`[[role]]` to |options|["{{SFrameTransformOptions/role}}"]. 6. Set |this|.`[[readable]]` to |this|.`[[transform]]`.`[[readable]]`. 7. Set |this|.`[[writable]]` to |this|.`[[transform]]`.`[[writable]]`. +7. Set |this|.`[[useSFrame]]` to true. ## Algorithm ## {#sframe-transform-algorithm} The SFrame transform algorithm, given |sframe| as a SFrameTransform object and |frame|, runs these steps: 1. Let |role| be |sframe|.`[[role]]`. -1. If |frame|.`[[owner]]` is a {{RTCRtpSender}}, set |role| to 'encrypt'. -1. If |frame|.`[[owner]]` is a {{RTCRtpReceiver}}, set |role| to 'decrypt'. +1. If |sframe|.`[[owner]]` is a {{RTCRtpSender}}, set |role| to 'encrypt'. +1. If |sframe|.`[[owner]]` is a {{RTCRtpReceiver}}, set |role| to 'decrypt'. +1. If |sframe|.`[[owner]]` is a {{RTCRtpReceiver}} and |frame|.`[[useSFrame]]` is not true, [=queue a task=] to run the following steps: + 1. [=fire an event=] named {{SFrameTransform/onerror|error}} at |sframe|, + using the {{SFrameTransformErrorEvent}} interface with its {{SFrameTransformErrorEvent/errorType}} attribute set to {{SFrameTransformErrorEventType/packetization}} + and its {{SFrameTransformErrorEvent/frame}} attribute set to |frame|. + 1. Abort these steps. 1. Let |data| be undefined. 1. If |frame| is a {{BufferSource}}, set |data| to |frame|. 1. If |frame| is a {{RTCEncodedAudioFrame}}, set |data| to |frame|.{{RTCEncodedAudioFrame/data}} @@ -302,6 +327,7 @@ The SFrame transform algorithm, given |sframe| as a SFrameTransform object and | 1. If |frame| is a {{BufferSource}}, set |frame| to |buffer|. 1. If |frame| is a {{RTCEncodedAudioFrame}}, set |frame|.{{RTCEncodedAudioFrame/data}} to |buffer|. 1. If |frame| is a {{RTCEncodedVideoFrame}}, set |frame|.{{RTCEncodedVideoFrame/data}} to |buffer|. +1. Set |frame|.`[[useSFrame]]` to true. 1. [=ReadableStream/Enqueue=] |frame| in |sframe|.`[[transform]]`. ## Methods ## {#sframe-transform-methods} @@ -314,6 +340,27 @@ The setEncryptionKey(|key|, |keyID|) met 3. [=Resolve=] |promise| with undefined. 4. Return |promise|. +## SFrame packetization integration ## {#sframe-packetization} + +A SFrame packetizer is responsible to generate SFrame packets from media content. +In the context of this specification, the [=SFrame packetizer=] is not responsible for doing the actual encryption. +Instead, the transform is responsible for doing so. The [=SFrame packetizer=] is responsible for splitting +SFrame frames as needed so that they fit in RTP packets. + +Similarly, a SFrame depacketizer is responsible to assemble RTP packets into a complete SFrame frame. +It is not responsible for doing the actual decryption, the transform is responsible for doing so. + +WebRTC encoded transform model is a per frame processing. SFrame can either be applied on each frame or on subframes. +WebRTC encoded transform model is naturally aligned with aplying SFrame on a frame as a whole. +To preserve WebRTC encoded transform model when applying SFrame on subframes, the following conceptual steps can be done: +1. On sending side, the {{SFrameTransform}} may first split the media frame in subframes like would do a regular media packetizer, + and apply the SFrame encryption on each subframe. It then concatenates the encrypted subframes as a unique encrypted frame. + The transform provides the encrypted frame and information of each subframe so that the [=SFrame packetizer=] + generates individual packets for each subframe. +2. On receiving side, the [=SFrame depacketizer=] assembles all individual subframe RTP packets as a unique encrypted frame. + It is responsible to give the necessary subframe information to the transform so that the transform can apply the SFrame decryption + on each individual subframe contained in the unique encrypted frame and concatenate each decrypted subframe as a unique decrypted media frame. + If decryption of a single subframe fails, the whole encrypted frame is discarded. # RTCRtpScriptTransform # {#scriptTransform} @@ -881,9 +928,13 @@ interface RTCRtpScriptTransformer : EventTarget { readonly attribute any options; }; +enum RTCRtpScriptTransformType { + "sframe" +}; + [Exposed=Window] interface RTCRtpScriptTransform { - constructor(Worker worker, optional any options, optional sequence<object> transfer); + constructor(Worker worker, optional any options, optional sequence<object> transfer, optional RTCRtpScriptTransformType type); }; [Exposed=DedicatedWorker] @@ -895,15 +946,16 @@ interface KeyFrameRequestEvent : Event { ## Operations ## {#RTCRtpScriptTransform-operations} -The new RTCRtpScriptTransform(|worker|, |options|, |transfer|) constructor steps are: +The new RTCRtpScriptTransform(|worker|, |options|, |transfer|, |type|) constructor steps are: 1. Set |t1| to an [=identity transform stream=]. -2. Set |t2| to an [=identity transform stream=]. -3. Set |this|.`[[writable]]` to |t1|.`[[writable]]`. -4. Set |this|.`[[readable]]` to |t2|.`[[readable]]`. -5. Let |serializedOptions| be the result of [$StructuredSerializeWithTransfer$](|options|, |transfer|). -6. Let |serializedReadable| be the result of [$StructuredSerializeWithTransfer$](|t1|.`[[readable]]`, « |t1|.`[[readable]]` »). -7. Let |serializedWritable| be the result of [$StructuredSerializeWithTransfer$](|t2|.`[[writable]]`, « |t2|.`[[writable]]` »). -8. [=Queue a task=] on the DOM manipulation [=task source=] |worker|'s global scope to run the following steps: +1. Set |t2| to an [=identity transform stream=]. +1. Set |this|.`[[writable]]` to |t1|.`[[writable]]`. +1. Set |this|.`[[readable]]` to |t2|.`[[readable]]`. +1. If |type| is equal to {{RTCRtpScriptTransformType/"sframe"}}, set |this|.`[[useSFrame]]` to true. +1. Let |serializedOptions| be the result of [$StructuredSerializeWithTransfer$](|options|, |transfer|). +1. Let |serializedReadable| be the result of [$StructuredSerializeWithTransfer$](|t1|.`[[readable]]`, « |t1|.`[[readable]]` »). +1. Let |serializedWritable| be the result of [$StructuredSerializeWithTransfer$](|t2|.`[[writable]]`, « |t2|.`[[writable]]` »). +1. [=Queue a task=] on the DOM manipulation [=task source=] |worker|'s global scope to run the following steps: 1. Let |transformerOptions| be the result of [$StructuredDeserializeWithTransfer$](|serializedOptions|, the current Realm). 2. Let |readable| be the result of [$StructuredDeserializeWithTransfer$](|serializedReadable|, the current Realm). 3. Let |writable| be the result of [$StructuredDeserializeWithTransfer$](|serializedWritable|, the current Realm).