Automatic pairing

[backkem]

This ticket explores means of pairing automatically without user interaction. These techniques are meant to be additive to the core pairing logic to enable user-friendliness improvements. Therefore, they don't have to be adhere to the LAN-only prerequisites that the main spec but they cannot compromise security.
Potential approaches include:

• Pre-shared keys
• External signaling, E.g. using a cloud service.

[guest271314]

External signaling, E.g. using a cloud service.

That is absolutely not 'local'. Signaling can be done entirely locally, e.g., https://plnkr.co/edit/1HsvQh08tYb24810?preview.

[backkem]

First off, maybe the term 'local' is a bit confusing. It's meant to include a whole LAN, not just local loopback. The spec tries to design for making connections between two machines on the same LAN, without WAN support (by default). It also assumes the LAN is hostile. Therefore, there is a pairing step, like a chromecast or bluetooth have, for example.

This ticket is indeed out of that initial scope. However, it means to explore if we can have hooks into the pairing step, that would allow implementers to do pairing over existing connections, such as the WAN. It is meant to be completely optional, just to improve user experience.

[guest271314]

I have not tried Chromecast.

What you describe appears to be similar to how when permission is granted to a site for local devices via navigator.mediaDevices.getUserMedia() the site can retain that permission unless explicitly revoked.

Now, getUserMedia() is specified to only capture microphone and camera, however, at the local machine the user can create virtual devices, e.g., a video stream that is exposed to the site as a web camera

$ sudo modprobe v4l2loopback video_nr="4" 'card_label=Virtual_Camera' exclusive_caps=1 max_buffers=2

$ ffmpeg -stream_loop -1 -re -i blade_runner.webm -r 30 -f v4l2 -vcodec rawvideo -pix_fmt rgb24 /dev/video4

then get the video from 'Virtual_Camera' via getUserMedia({video: true}).

or a local file or live entire system audio output guest271314/SpeechSynthesisRecorder#17

$ pactl load-module module-combine-sink \
sink_name=Web_Speech_Sink slaves=$(pacmd list-sinks | grep -A1 "* index" | grep -oP "<\K[^ >]+") \
sink_properties=device.description="Web_Speech_Stream" \
format=s16le \
channels=1 \
rate=22050

$ pactl move-sink-input $(pacmd list-sink-inputs | tac | perl -E'undef$/;$_=<>;/speech-dispatcher-espeak-ng.*?index: (\d+)\n/s;say $1') Web_Speech_Sink

$ pactl load-module module-remap-source \
master=Web_Speech_Sink.monitor \
source_name=Web_Speech_Monitor \
source_properties=device.description=Web_Speech_Output

or specific sink-input for capture https://github.com/guest271314/setUserMediaAudioSource.

Thus, if I gather what you are trying to do here, i.e., "cloud service" can be a local Raspberry Pi, the specification part re "pairing" and persistence can be "borrowed" from Media Capture and Streams specification, to the extent application.

[backkem]

Indeed, I'm trying to maximally re-use existing work to increase the chances of this making the cut. As you mentioned, the JS API is inspired by GetUserMedia and WebTransport. At the protocol level, almost everything is based on the work done for the Open Screen Protocol.

(Temporary spec is here, I didn't get around to porting it to Git yet)

[guest271314]

Yes, I located and read the draft specification. There are obvious overlapping interests with this proposal httpslocal/proposals#2.

I suggest that you contact the relevant stakeholders there so that you can merge interests into a single proposal.

I have filed multiple specification and implementer issues re creation and access to local hardware and virtual devices, again, a brief synopsis can be found at https://github.com/guest271314/captureSystemAudio#references. TL;DR. Beware, I am verbose, yet very thorough. In pertinent part see w3c/mediacapture-main#650 and w3c/mediacapture-main#654. In theory this proposal could be incorporated into Extensions to Media Capture and Streams by the WebRTC Working Group https://github.com/w3c/mediacapture-extensions, yet you could more than likely lose some independence. See also https://w3c.github.io/mediacapture-automation/.