Skip to content

Add SFrame packetization handling for SFrameTransform #252

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add SFrame packetization handling for SFrameTransform #252

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 64 additions & 12 deletions index.bs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,9 @@ spec:webidl; type:dfn; text:resolve
<pre class=link-defaults>
spec:streams; type:interface; text:ReadableStream
</pre>
<pre class=anchors>
url: https://w3c.github.io/webrtc-pc/#dfn-update-the-negotiation-needed-flag; text: update the negotiation-needed flag; type: dfn; spec: WEBRTC
</pre>

# Introduction # {#introduction}

Expand Down Expand Up @@ -138,6 +141,7 @@ The <dfn abstract-op>readEncodedData</dfn> algorithm is given a |rtcObject| as p
1. Increment |rtcObject|.`[[lastEnqueuedFrameCounter]]` by <code>1</code>.
1. Let |frame| be the newly produced frame.
1. Set |frame|.`[[owner]]` to |rtcObject|.
1. Set |this|.`[[writable]]` to |this|.`[[transform]]`.`[[writable]]`.
1. Set |frame|.`[[counter]]` to |rtcObject|.`[[lastEnqueuedFrameCounter]]`.
1. If the frame has been produced by a {{RTCRtpReceiver}}:
1. If the relevant RTP packet contains the
Expand All @@ -150,6 +154,7 @@ The <dfn abstract-op>readEncodedData</dfn> algorithm is given a |rtcObject| as p
[[RTP-EXT-CAPTURE-TIME#timestamp-interpolation|timestamp interpolation]] and set |frame|.`[[senderCaptureTimeOffset]]`
to the most recent value that was present.
1. Otherwise, set |frame|.`[[captureTime]]` to undefined and set |frame|.`[[senderCaptureTimeOffset]]` to undefined.
1. If |frame| was produced by a [=SFrame depacketizer=], set |frame|.`[[useSFrame]]` to true.
1. If the frame has been produced by a {{RTCRtpSender}}, set |frame|.`[[captureTime]]` to the capture timestamp
using the methodology described in [[RTP-EXT-CAPTURE-TIME#absolute-capture-timestamp]] and set frame.`[[senderCaptureTimeOffset]]`
to undefined.
Expand All @@ -162,8 +167,10 @@ The <dfn abstract-op>writeEncodedData</dfn> algorithm is given a |rtcObject| as
1. Let |data| be |frame|.`[[data]]`.
1. Let |serializedFrame| be [$StructuredSerializeWithTransfer$](|frame|, « |data| »).
1. Let |frameCopy| be [$StructuredDeserializeWithTransfer$](|serializedFrame|, |frame|'s [=relevant realm=]).
1. If |frame|.`[[useSFrame]]` is true, set |frameCopy|.`[[useSFrame]]` to true.
1. Enqueue |frameCopy| for processing as if it came directly from the encoded data source, by running one of the following steps:
* If |rtcObject| is a {{RTCRtpSender}}, enqueue |frameCopy| to |rtcObject|'s packetizer, to be processed [=in parallel=].
If |frameCopy|.`[[useSFrame]]` is true, |rtcObject|'s MUST use a [=SFrame packetizer=] or skip processing of |frameCopy|.
* If |rtcObject| is a {{RTCRtpReceiver}}, enqueue |frameCopy| it to |rtcObject|'s decoder, to be processed [=in parallel=].
1. Return [=a promise resolved with=] undefined.

Expand All @@ -181,6 +188,7 @@ A RTCRtpTransform has private slots:
* `[[readable]]` of type {{ReadableStream}}.
* `[[writable]]` of type {{WritableStream}}.
* `[[owner]]` of type {{RTCRtpSender}} or {{RTCRtpReceiver}}, initialized to null.
* `[[useSFrame]]` of type boolean. // FIXME: Decide whether augmenting this boolean with either cipher suite support or whether doing frame vs. packet based encryption.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we do the FIXME as a note so it shows up in the spec?


Each RTCRtpTransform has an <dfn abstract-op for=RTCRtpTransform>association steps</dfn> set, which is empty by default.

Expand All @@ -199,6 +207,16 @@ The `transform` setter steps are:
1. [=AbortSignal/Add=] the [$chain transform algorithm$] to [=this=].`[[pipeToController]]`'s [=AbortController/signal=].
2. [=AbortController/signal abort=] on [=this=].`[[pipeToController]]`.
1. Else, run the [$chain transform algorithm$] steps.
1. If [=this=] is a {{RTCRtpSender}}, run the following substeps:
1. Let |useSFrame| be true if [=this=] is configured to use a [=SFrame packetizer=] and false otherwise.
1. If |useSFrame| is equal to |checkedTransform|.`[[useSFrame]]`, abort these substeps.
1. Configure [=this=]'s packetizer to use SFrame if |checkedTransform|.`[[useSFrame]]` is true and to not use SFrame if |checkedTransform|.`[[useSFrame]]` is false.
1. [=Update the negotiation-needed flag=] for [=this=]'s connection.
1. Otherwise, run the following steps:
1. Let |useSFrame| be true if [=this=] is configured to use a [=SFrame depacketizer=] and false otherwise.
1. If |useSFrame| is equal to |checkedTransform|.`[[useSFrame]]`, abort these substeps.
1. Configure [=this=]'s depacketizer to use SFrame if |checkedTransform|.`[[useSFrame]]` is true and to not use SFrame if |checkedTransform|.`[[useSFrame]]` is false.
1. [=Update the negotiation-needed flag=] for [=this=]'s connection.
1. Set [=this=].`[[pipeToController]]` to |newPipeToController|.
1. Set [=this=].`[[transform]]` to |transform|.
1. Run the steps in the set of [$association steps$] of |transform| with [=this=].
Expand Down Expand Up @@ -248,7 +266,8 @@ SFrameTransform includes GenericTransformStream;
enum SFrameTransformErrorEventType {
"authentication",
"keyID",
"syntax"
"syntax",
"packetization"
};

[Exposed=(Window,DedicatedWorker)]
Expand All @@ -275,13 +294,19 @@ The <dfn constructor for="SFrameTransform" lt="SFrameTransform(options)"><code>n
5. Set |this|.`[[role]]` to |options|["{{SFrameTransformOptions/role}}"].
6. Set |this|.`[[readable]]` to |this|.`[[transform]]`.`[[readable]]`.
7. Set |this|.`[[writable]]` to |this|.`[[transform]]`.`[[writable]]`.
7. Set |this|.`[[useSFrame]]` to true.

## Algorithm ## {#sframe-transform-algorithm}

The SFrame transform algorithm, given |sframe| as a SFrameTransform object and |frame|, runs these steps:
1. Let |role| be |sframe|.`[[role]]`.
1. If |frame|.`[[owner]]` is a {{RTCRtpSender}}, set |role| to 'encrypt'.
1. If |frame|.`[[owner]]` is a {{RTCRtpReceiver}}, set |role| to 'decrypt'.
1. If |sframe|.`[[owner]]` is a {{RTCRtpSender}}, set |role| to 'encrypt'.
1. If |sframe|.`[[owner]]` is a {{RTCRtpReceiver}}, set |role| to 'decrypt'.
1. If |sframe|.`[[owner]]` is a {{RTCRtpReceiver}} and |frame|.`[[useSFrame]]` is not true, [=queue a task=] to run the following steps:
1. [=fire an event=] named {{SFrameTransform/onerror|error}} at |sframe|,
using the {{SFrameTransformErrorEvent}} interface with its {{SFrameTransformErrorEvent/errorType}} attribute set to {{SFrameTransformErrorEventType/packetization}}
and its {{SFrameTransformErrorEvent/frame}} attribute set to |frame|.
1. Abort these steps.
1. Let |data| be undefined.
1. If |frame| is a {{BufferSource}}, set |data| to |frame|.
1. If |frame| is a {{RTCEncodedAudioFrame}}, set |data| to |frame|.{{RTCEncodedAudioFrame/data}}
Expand All @@ -302,6 +327,7 @@ The SFrame transform algorithm, given |sframe| as a SFrameTransform object and |
1. If |frame| is a {{BufferSource}}, set |frame| to |buffer|.
1. If |frame| is a {{RTCEncodedAudioFrame}}, set |frame|.{{RTCEncodedAudioFrame/data}} to |buffer|.
1. If |frame| is a {{RTCEncodedVideoFrame}}, set |frame|.{{RTCEncodedVideoFrame/data}} to |buffer|.
1. Set |frame|.`[[useSFrame]]` to true.
1. [=ReadableStream/Enqueue=] |frame| in |sframe|.`[[transform]]`.

## Methods ## {#sframe-transform-methods}
Expand All @@ -314,6 +340,27 @@ The <dfn method for="SFrameTransform">setEncryptionKey(|key|, |keyID|)</dfn> met
3. [=Resolve=] |promise| with undefined.
4. Return |promise|.

## SFrame packetization integration ## {#sframe-packetization}

A <dfn>SFrame packetizer</dfn> is responsible to generate SFrame packets from media content.
In the context of this specification, the [=SFrame packetizer=] is not responsible for doing the actual encryption.
Instead, the transform is responsible for doing so. The [=SFrame packetizer=] is responsible for splitting
SFrame frames as needed so that they fit in RTP packets.

Similarly, a <dfn>SFrame depacketizer</dfn> is responsible to assemble RTP packets into a complete SFrame frame.
It is not responsible for doing the actual decryption, the transform is responsible for doing so.

WebRTC encoded transform model is a per frame processing. SFrame can either be applied on each frame or on subframes.
WebRTC encoded transform model is naturally aligned with aplying SFrame on a frame as a whole.
To preserve WebRTC encoded transform model when applying SFrame on subframes, the following conceptual steps can be done:
1. On sending side, the {{SFrameTransform}} may first split the media frame in subframes like would do a regular media packetizer,
and apply the SFrame encryption on each subframe. It then concatenates the encrypted subframes as a unique encrypted frame.
The transform provides the encrypted frame and information of each subframe so that the [=SFrame packetizer=]
generates individual packets for each subframe.
2. On receiving side, the [=SFrame depacketizer=] assembles all individual subframe RTP packets as a unique encrypted frame.
It is responsible to give the necessary subframe information to the transform so that the transform can apply the SFrame decryption
on each individual subframe contained in the unique encrypted frame and concatenate each decrypted subframe as a unique decrypted media frame.
If decryption of a single subframe fails, the whole encrypted frame is discarded.

# RTCRtpScriptTransform # {#scriptTransform}

Expand Down Expand Up @@ -881,9 +928,13 @@ interface RTCRtpScriptTransformer : EventTarget {
readonly attribute any options;
};

enum RTCRtpScriptTransformType {
"sframe"
};

[Exposed=Window]
interface RTCRtpScriptTransform {
constructor(Worker worker, optional any options, optional sequence&lt;object&gt; transfer);
constructor(Worker worker, optional any options, optional sequence&lt;object&gt; transfer, optional RTCRtpScriptTransformType type);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I'd prefer to not add more arguments to the constructor but instead reuse the options object we already overload to send to worker. I'm not super worried about name collisions.

};

[Exposed=DedicatedWorker]
Expand All @@ -895,15 +946,16 @@ interface KeyFrameRequestEvent : Event {

## Operations ## {#RTCRtpScriptTransform-operations}

The <dfn constructor for="RTCRtpScriptTransform" lt="RTCRtpScriptTransform(worker, options)"><code>new RTCRtpScriptTransform(|worker|, |options|, |transfer|)</code></dfn> constructor steps are:
The <dfn constructor for="RTCRtpScriptTransform" lt="RTCRtpScriptTransform(worker, options)"><code>new RTCRtpScriptTransform(|worker|, |options|, |transfer|, |type|)</code></dfn> constructor steps are:
1. Set |t1| to an [=identity transform stream=].
2. Set |t2| to an [=identity transform stream=].
3. Set |this|.`[[writable]]` to |t1|.`[[writable]]`.
4. Set |this|.`[[readable]]` to |t2|.`[[readable]]`.
5. Let |serializedOptions| be the result of [$StructuredSerializeWithTransfer$](|options|, |transfer|).
6. Let |serializedReadable| be the result of [$StructuredSerializeWithTransfer$](|t1|.`[[readable]]`, « |t1|.`[[readable]]` »).
7. Let |serializedWritable| be the result of [$StructuredSerializeWithTransfer$](|t2|.`[[writable]]`, « |t2|.`[[writable]]` »).
8. [=Queue a task=] on the DOM manipulation [=task source=] |worker|'s global scope to run the following steps:
1. Set |t2| to an [=identity transform stream=].
1. Set |this|.`[[writable]]` to |t1|.`[[writable]]`.
1. Set |this|.`[[readable]]` to |t2|.`[[readable]]`.
1. If |type| is equal to {{RTCRtpScriptTransformType/"sframe"}}, set |this|.`[[useSFrame]]` to true.
1. Let |serializedOptions| be the result of [$StructuredSerializeWithTransfer$](|options|, |transfer|).
1. Let |serializedReadable| be the result of [$StructuredSerializeWithTransfer$](|t1|.`[[readable]]`, « |t1|.`[[readable]]` »).
1. Let |serializedWritable| be the result of [$StructuredSerializeWithTransfer$](|t2|.`[[writable]]`, « |t2|.`[[writable]]` »).
1. [=Queue a task=] on the DOM manipulation [=task source=] |worker|'s global scope to run the following steps:
1. Let |transformerOptions| be the result of [$StructuredDeserializeWithTransfer$](|serializedOptions|, the current Realm).
2. Let |readable| be the result of [$StructuredDeserializeWithTransfer$](|serializedReadable|, the current Realm).
3. Let |writable| be the result of [$StructuredDeserializeWithTransfer$](|serializedWritable|, the current Realm).
Expand Down