forked from webrtc/samples
-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add video processing in worker sample
- Loading branch information
Showing
8 changed files
with
296 additions
and
17 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
27 changes: 27 additions & 0 deletions
27
src/content/insertable-streams/video-processing-worker/css/main.css
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| /* | ||
| * Copyright (c) 2015 The WebRTC project authors. All Rights Reserved. | ||
| * | ||
| * Use of this source code is governed by a BSD-style license | ||
| * that can be found in the LICENSE file in the root of the source | ||
| * tree. | ||
| */ | ||
|
|
||
| button { | ||
| margin: 20px 10px 0 0; | ||
| width: 100px; | ||
| } | ||
|
|
||
| div#buttons { | ||
| margin: 0 0 20px 0; | ||
| } | ||
|
|
||
| div#status { | ||
| height: 2em; | ||
| margin: 1em 0 0 0; | ||
| } | ||
|
|
||
| video { | ||
| --width: 45%; | ||
| width: var(--width); | ||
| height: calc(var(--width) * 0.75); | ||
| } |
82 changes: 82 additions & 0 deletions
82
src/content/insertable-streams/video-processing-worker/index.html
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| <!DOCTYPE html> | ||
| <!-- | ||
| * Copyright (c) 2021 The WebRTC project authors. All Rights Reserved. | ||
| * | ||
| * Use of this source code is governed by a BSD-style license | ||
| * that can be found in the LICENSE file in the root of the source | ||
| * tree. | ||
| --> | ||
| <html> | ||
| <head> | ||
|
|
||
| <meta charset="utf-8"> | ||
| <meta name="description" content="WebRTC code samples"> | ||
| <meta name="viewport" content="width=device-width, user-scalable=yes, initial-scale=1, maximum-scale=1"> | ||
| <meta itemprop="description" content="Client-side WebRTC code samples"> | ||
| <meta itemprop="image" content="../../../images/webrtc-icon-192x192.png"> | ||
| <meta itemprop="name" content="WebRTC code samples"> | ||
| <meta name="mobile-web-app-capable" content="yes"> | ||
| <meta id="theme-color" name="theme-color" content="#ffffff"> | ||
|
|
||
| <base target="_blank"> | ||
|
|
||
| <title>Insertable Streams - Video Processing in a worker</title> | ||
|
|
||
| <link rel="icon" sizes="192x192" href="../../../images/webrtc-icon-192x192.png"> | ||
| <link href="//fonts.googleapis.com/css?family=Roboto:300,400,500,700" rel="stylesheet" type="text/css"> | ||
| <link rel="stylesheet" href="../../../css/main.css"/> | ||
| <link rel="stylesheet" href="css/main.css"/> | ||
|
|
||
| </head> | ||
|
|
||
| <body> | ||
|
|
||
| <div id="container"> | ||
| <h1><a href="//webrtc.github.io/samples/" title="WebRTC samples homepage">WebRTC samples</a> | ||
| <span>Breakout Box video processing in worker</span></h1> | ||
|
|
||
| <p>This sample shows how to perform processing on a video stream using the experimental | ||
| <a href="https://github.com/w3c/mediacapture-transform">mediacapture-transform</a> API | ||
| in a Worker. | ||
| </p> | ||
|
|
||
| <video id="localVideo" playsinline autoplay muted></video> | ||
| <video id="croppedVideo" playsinline autoplay muted></video> | ||
|
|
||
| <div> | ||
| <span>Transform:</span> | ||
| <select id="transformSelector"> | ||
| <option selected value="webgl-background-blur">WebGL background blur</option> | ||
| <option value="webgpu-background-blur">WebGPU/WebNN background blur</option> | ||
| <option value="webgl">WebGL wrap</option> | ||
| <option value="canvas2d">Canvas2D</option> | ||
| <option value="noop">Do nothing</option> | ||
| <option value="drop">Drop frames at random</option> | ||
| <option value="delay">Delay all frames by 100ms</option> | ||
| <option value="webcodec">Run frames through WebCodec</option> | ||
| </select> | ||
| </div> | ||
|
|
||
| <div class="box"> | ||
| <button id="startButton">Start</button> | ||
| <button id="stopButton" disabled>Stop</button> | ||
| </div> | ||
|
|
||
| <p> | ||
| <b>Note</b>: This sample is using an experimental API that has not yet been standardized. As | ||
| of 2021-07-16, this API is available in Chrome M91 if the experimental code is enabled on | ||
| the command line with | ||
| <code>--enable-blink-features=MediaStreamInsertableStreams</code>. | ||
| </p> | ||
| <a href="https://github.com/huningxin/webrtc-samples/tree/background_blur/src/content/insertable-streams/video-processing-worker" | ||
| title="View source for this page on GitHub" id="viewSource">View source on GitHub</a> | ||
|
|
||
| </div> | ||
|
|
||
| <script src="https://webrtc.github.io/adapter/adapter-latest.js"></script> | ||
| <script src="../../../js/third_party/stats.min.js"></script> | ||
| <script src="js/main.js" async></script> | ||
|
|
||
| <script src="../../../js/lib/ga.js"></script> | ||
| </body> | ||
| </html> |
70 changes: 70 additions & 0 deletions
70
src/content/insertable-streams/video-processing-worker/js/main.js
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| /* | ||
| * Copyright (c) 2021 The WebRTC project authors. All Rights Reserved. | ||
| * | ||
| * Use of this source code is governed by a BSD-style license | ||
| * that can be found in the LICENSE file in the root of the source | ||
| * tree. | ||
| */ | ||
|
|
||
| 'use strict'; | ||
|
|
||
| /* global MediaStreamTrackProcessor, MediaStreamTrackGenerator */ | ||
| if (typeof MediaStreamTrackProcessor === 'undefined' || | ||
| typeof MediaStreamTrackGenerator === 'undefined') { | ||
| alert( | ||
| 'Your browser does not support the experimental MediaStreamTrack API ' + | ||
| 'for Insertable Streams of Media. See the note at the bottom of the ' + | ||
| 'page.'); | ||
| } | ||
|
|
||
| const startButton = document.getElementById('startButton'); | ||
| const stopButton = document.getElementById('stopButton'); | ||
| const localVideo = document.getElementById('localVideo'); | ||
| const croppedVideo = document.getElementById('croppedVideo'); | ||
| const transformSelector = document.getElementById('transformSelector'); | ||
|
|
||
| const stats = new Stats(); | ||
| stats.showPanel(0); // 0: fps, 1: ms, 2: mb, 3+: custom | ||
| document.body.appendChild(stats.dom); | ||
| const updateFPS = (now, metadata) => { | ||
| stats.update(); | ||
| croppedVideo.requestVideoFrameCallback(updateFPS); | ||
| }; | ||
| croppedVideo.requestVideoFrameCallback(updateFPS); | ||
|
|
||
| const worker = new Worker('./js/worker.js', {name: 'Video processing worker'}); | ||
| let stream = null; | ||
| startButton.addEventListener('click', async () => { | ||
| stream = await navigator.mediaDevices.getUserMedia({audio: false, video: true}); | ||
| localVideo.srcObject = stream; | ||
|
|
||
| const [track] = stream.getTracks(); | ||
| const processor = new MediaStreamTrackProcessor({track}); | ||
| const {readable} = processor; | ||
|
|
||
| const generator = new MediaStreamTrackGenerator({kind: 'video'}); | ||
| const {writable} = generator; | ||
| croppedVideo.srcObject = new MediaStream([generator]); | ||
|
|
||
| worker.postMessage({ | ||
| operation: 'start', | ||
| transformType: transformSelector.value, | ||
| readable, | ||
| writable, | ||
| }, [readable, writable]); | ||
| stopButton.disabled = false; | ||
| startButton.disabled = true; | ||
| }); | ||
|
|
||
| stopButton.addEventListener('click', async () => { | ||
| localVideo.pause(); | ||
| localVideo.srcObject = null; | ||
| croppedVideo.pause(); | ||
| croppedVideo.srcObject = null; | ||
| if (stream) { | ||
| stream.getTracks().forEach(t => t.stop()); | ||
| } | ||
| worker.postMessage({operation: 'stop'}); | ||
| stopButton.disabled = true; | ||
| startButton.disabled = false; | ||
| }); |
74 changes: 74 additions & 0 deletions
74
src/content/insertable-streams/video-processing-worker/js/worker.js
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| /* | ||
| * Copyright (c) 2021 The WebRTC project authors. All Rights Reserved. | ||
| * | ||
| * Use of this source code is governed by a BSD-style license | ||
| * that can be found in the LICENSE file in the root of the source | ||
| * tree. | ||
| */ | ||
| importScripts('https://cdn.jsdelivr.net/npm/@tensorflow/tfjs/dist/tf.min.js') | ||
| importScripts('../../video-processing/js/webgl-background-blur.js'); | ||
| importScripts('../../video-processing/js/webgpu-background-blur.js'); | ||
| importScripts('../../video-processing/js/canvas-transform.js'); | ||
| importScripts('../../video-processing/js/simple-transforms.js'); | ||
| importScripts('../../video-processing/js/webcodec-transform.js'); | ||
| importScripts('../../video-processing/js/webgl-transform.js/'); | ||
| importScripts('../../video-processing/js/webnn-deeplabv3.js'); | ||
| importScripts('../../../../js/third_party/numpy.js'); | ||
|
|
||
| 'use strict'; | ||
|
|
||
| let frameTransform = null; | ||
|
|
||
| async function transform(frame, controller) { | ||
| if (frameTransform) { | ||
| await frameTransform.transform(frame, controller); | ||
| } | ||
| } | ||
|
|
||
| onmessage = async (event) => { | ||
| const {operation, transformType} = event.data; | ||
| if (operation === 'start') { | ||
| switch (transformType) { | ||
| case 'webgl': | ||
| frameTransform = new WebGLTransform(); | ||
| break; | ||
| case 'webgl-background-blur': | ||
| frameTransform = new WebGLBackgroundBlurTransform(); | ||
| break; | ||
| case 'webgpu-background-blur': | ||
| frameTransform = new WebGPUBackgroundBlurTransform(); | ||
| break; | ||
| case 'canvas2d': | ||
| frameTransform = new CanvasTransform(); | ||
| break; | ||
| case 'drop': | ||
| // Defined in simple-transforms.js. | ||
| frameTransform = new DropTransform(); | ||
| break; | ||
| case 'noop': | ||
| // Defined in simple-transforms.js. | ||
| frameTransform = new NullTransform(); | ||
| break; | ||
| case 'delay': | ||
| // Defined in simple-transforms.js. | ||
| frameTransform = new DelayTransform(); | ||
| break; | ||
| case 'webcodec': | ||
| // Defined in webcodec-transform.js | ||
| frameTransform = new WebCodecTransform(); | ||
| break; | ||
| default: | ||
| throw new Error(`unknown transform ${transformType}`); | ||
| break; | ||
| } | ||
| frameTransform.init(); | ||
| const {readable, writable} = event.data; | ||
| readable | ||
| .pipeThrough(new TransformStream({transform})) | ||
| .pipeTo(writable); | ||
| } else if (operation === 'stop') { | ||
| frameTransform.destroy(); | ||
| } else { | ||
| throw new Error(`unknown operation ${operation}`); | ||
| } | ||
| }; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters