Update video files to improve external_texture* related cts coverage (g…

…puweb#3245)

* Update video files to improve external_texture* related cts coverage

This PR updated video files by referring webgl related cases here:
https://github.com/KhronosGroup/WebGL/blob/main/sdk/tests/conformance/textures/misc/video-rotation.html#L112

The PR changes:
- README.md with latest cmds to generate test video files.
- Rotated video content.
- Update video expected pixels values based on new test video files.
- Related cts logic changes, including removing magic numbers.

Fix:gpuweb#3232

* fix lint issue and address comments

* Update video files, test cases and comments to describe transformation
applies order.

* reduce video size by encoding less frames

src/resources/README.md

@@ -2,14 +2,64 @@ Always use `getResourcePath()` to get the appropriate path to these resources de
 on the context (WPT, standalone, worker, etc.)
 
 
-The test video files were generated with the ffmpeg cmds below:
-ffmpeg.exe -loop 1 -i .\four-colors.png -c:v libvpx -pix_fmt yuv420p -frames 500 -colorspace smpte170m -color_primaries smpte170m -color_trc smpte170m -color_range tv four-colors-vp8-bt601.webm
-ffmpeg.exe -loop 1 -i .\four-colors.png -c:v libtheora -pix_fmt yuv420p -frames 500 -colorspace smpte170m -color_primaries smpte170m -color_trc smpte170m -color_range tv four-colors-theora-bt601.ogv
-ffmpeg.exe -loop 1 -i .\four-colors.png -c:v libx264 -pix_fmt yuv420p -frames 500 -colorspace smpte170m -color_primaries smpte170m -color_trc smpte170m -color_range tv four-colors-h264-bt601.mp4
-ffmpeg.exe -loop 1 -i .\four-colors.png -c:v libvpx-vp9 -pix_fmt yuv420p -frames 500 -colorspace smpte170m -color_primaries smpte170m -color_trc smpte170m -color_range tv four-colors-vp9-bt601.webm
-ffmpeg.exe -loop 1 -i .\four-colors.png -c:v libvpx-vp9 -pix_fmt yuv420p -frames 500 -colorspace bt709 -color_primaries bt709 -color_trc bt709 -color_range tv -vf scale=out_color_matrix=bt709:out_range=tv four-colors-vp9-bt709.webm
-
-These rotation test files are copies of four-colors-h264-bt601.mp4 with metadata changes.
-ffmpeg.exe -i .\four-colors-h264-bt601.mp4 -c copy -metadata:s:v rotate=90 four-colors-h264-bt601-rotate-90.mp4
-ffmpeg.exe -i .\four-colors-h264-bt601.mp4 -c copy -metadata:s:v rotate=180 four-colors-h264-bt601-rotate-180.mp4
-ffmpeg.exe -i .\four-colors-h264-bt601.mp4 -c copy -metadata:s:v rotate=270 four-colors-h264-bt601-rotate-270.mp4
+The test video files were generated with by ffmpeg cmds below:
+```
+// Generate four-colors-vp8-bt601.webm, mimeType: 'video/webm; codecs=vp8'
+ffmpeg.exe -loop 1 -i .\four-colors.png -c:v libvpx -pix_fmt yuv420p -frames 50 -colorspace smpte170m -color_primaries smpte170m -color_trc smpte170m -color_range tv four-colors-vp8-bt601.webm
+
+// Generate four-colors-theora-bt601.ogv, mimeType: 'video/ogg; codecs=theora'
+ffmpeg.exe -loop 1 -i .\four-colors.png -c:v libtheora -pix_fmt yuv420p -frames 50 -colorspace smpte170m -color_primaries smpte170m -color_trc smpte170m -color_range tv four-colors-theora-bt601.ogv
+
+// Generate four-colors-h264-bt601.mp4, mimeType: 'video/mp4; codecs=h264'
+ffmpeg.exe -loop 1 -i .\four-colors.png -c:v libx264 -pix_fmt yuv420p -frames 50 -colorspace smpte170m -color_primaries smpte170m -color_trc smpte170m -color_range tv four-colors-h264-bt601.mp4
+
+// Generate four-colors-vp9-bt601.webm, mimeType: 'video/webm; codecs=vp9'
+ffmpeg.exe -loop 1 -i .\four-colors.png -c:v libvpx-vp9 -pix_fmt yuv420p -frames 50 -colorspace smpte170m -color_primaries smpte170m -color_trc smpte170m -color_range tv four-colors-vp9-bt601.webm
+
+// Generate four-colors-vp9-bt709.webm, mimeType: 'video/webm; codecs=vp9'
+ffmpeg.exe -loop 1 -i .\four-colors.png -c:v libvpx-vp9 -pix_fmt yuv420p -frames 50 -colorspace bt709 -color_primaries bt709 -color_trc bt709 -color_range tv -vf scale=out_color_matrix=bt709:out_range=tv four-colors-vp9-bt709.webm
+
+// Generate  four-colors-vp9-bt601.mp4, mimeType: 'video/mp4; codecs=vp9'
+ffmpeg.exe -loop 1 -i .\four-colors.png -c:v libvpx-vp9 -pix_fmt yuv420p -frames 50 -colorspace smpte170m -color_primaries smpte170m -color_trc smpte170m -color_range tv four-colors-vp9-bt601.mp4
+```
+
+Generate video files to test rotation behaviour.
+Use ffmepg to rotate video content x degrees in cw direction (by using `transpose`) and update transform matrix in metadata through `display_rotation` to x degrees to apply ccw direction rotation.
+
+H264 rotated video files are generated by ffmpeg cmds below:
+```
+// Generate four-colors-h264-bt601-rotate-90.mp4, mimeType: 'video/mp4; codecs=h264'
+ffmpeg.exe -loop 1 -i .\four-colors.png -c:v libx264 -pix_fmt yuv420p -frames 50 -colorspace smpte170m -color_primaries smpte170m -color_trc smpte170m -color_range tv -vf transpose=2 temp.mp4
+ffmpeg -display_rotation 270 -i temp.mp4 -c copy four-colors-h264-bt601-rotate-90.mp4
+rm temp.mp4
+
+// Generate four-colors-h264-bt601-rotate-180.mp4, mimeType: 'video/mp4; codecs=h264'
+ffmpeg.exe -loop 1 -i .\four-colors.png -c:v libx264 -pix_fmt yuv420p -frames 50 -colorspace smpte170m -color_primaries smpte170m -color_trc smpte170m -color_range tv -vf transpose=2,transpose=2 temp.mp4
+ffmpeg -display_rotation 180 -i temp.mp4 -c copy four-colors-h264-bt601-rotate-180.mp4
+rm temp.mp4
+
+// Generate four-colors-h264-bt601-rotate-270.mp4, mimeType: 'video/mp4; codecs=h264'
+ffmpeg.exe -loop 1 -i .\four-colors.png -c:v libx264 -pix_fmt yuv420p -frames 50 -colorspace smpte170m -color_primaries smpte170m -color_trc smpte170m -color_range tv -vf transpose=1 temp.mp4
+ffmpeg -display_rotation 90 -i temp.mp4 -c copy four-colors-h264-bt601-rotate-270.mp4
+rm temp.mp4
+
+```
+
+Vp9 rotated video files are generated by ffmpeg cmds below:
+```
+// Generate four-colors-h264-bt601-rotate-90.mp4, mimeType: 'video/mp4; codecs=vp9'
+ffmpeg.exe -loop 1 -i .\four-colors.png -c:v libvpx-vp9 -pix_fmt yuv420p -frames 50 -colorspace smpte170m -color_primaries smpte170m -color_trc smpte170m -color_range tv -vf transpose=2 temp.mp4
+ffmpeg -display_rotation 270 -i temp.mp4 -c copy four-colors-vp9-bt601-rotate-90.mp4
+rm temp.mp4
+
+// Generate four-colors-h264-bt601-rotate-180.mp4, mimeType: 'video/mp4; codecs=vp9'
+ffmpeg.exe -loop 1 -i .\four-colors.png -c:v libvpx-vp9 -pix_fmt yuv420p -frames 50 -colorspace smpte170m -color_primaries smpte170m -color_trc smpte170m -color_range tv -vf transpose=2,transpose=2 temp.mp4
+ffmpeg -display_rotation 180 -i temp.mp4 -c copy four-colors-vp9-bt601-rotate-180.mp4
+rm temp.mp4
+
+// Generate four-colors-h264-bt601-rotate-270.mp4, mimeType: 'video/mp4; codecs=vp9'
+ffmpeg.exe -loop 1 -i .\four-colors.png -c:v libvpx-vp9 -pix_fmt yuv420p -frames 50 -colorspace smpte170m -color_primaries smpte170m -color_trc smpte170m -color_range tv -vf transpose=1 temp.mp4
+ffmpeg -display_rotation 90 -i temp.mp4 -c copy four-colors-vp9-bt601-rotate-270.mp4
+rm temp.mp4
+
+```

src/webgpu/web_platform/copyToTexture/video.spec.ts

@@ -17,6 +17,7 @@ import {
   kPredefinedColorSpace,
   kVideoNames,
   kVideoInfo,
+  kVideoExpectedColors,
 } from '../../web_platform/util.js';
 
 const kFormat = 'rgba8unorm';
@@ -93,52 +94,56 @@ It creates HTMLVideoElement with videos under Resource folder.
         { width, height, depthOrArrayLayers: 1 }
       );
 
-      const expect = kVideoInfo[videoName].presentColors[dstColorSpace];
+      const srcColorSpace = kVideoInfo[videoName].colorSpace;
+      const presentColors = kVideoExpectedColors[srcColorSpace][dstColorSpace];
+
+      // visible rect is whole frame, no clipping.
+      const expect = kVideoInfo[videoName].display;
 
       if (srcDoFlipYDuringCopy) {
         t.expectSinglePixelComparisonsAreOkInTexture({ texture: dstTexture }, [
           // Flipped top-left.
           {
             coord: { x: width * 0.25, y: height * 0.25 },
-            exp: convertToUnorm8(expect.bottomLeftColor),
+            exp: convertToUnorm8(presentColors[expect.bottomLeftColor]),
           },
           // Flipped top-right.
           {
             coord: { x: width * 0.75, y: height * 0.25 },
-            exp: convertToUnorm8(expect.bottomRightColor),
+            exp: convertToUnorm8(presentColors[expect.bottomRightColor]),
           },
           // Flipped bottom-left.
           {
             coord: { x: width * 0.25, y: height * 0.75 },
-            exp: convertToUnorm8(expect.topLeftColor),
+            exp: convertToUnorm8(presentColors[expect.topLeftColor]),
           },
           // Flipped bottom-right.
           {
             coord: { x: width * 0.75, y: height * 0.75 },
-            exp: convertToUnorm8(expect.topRightColor),
+            exp: convertToUnorm8(presentColors[expect.topRightColor]),
           },
         ]);
       } else {
         t.expectSinglePixelComparisonsAreOkInTexture({ texture: dstTexture }, [
           // Top-left.
           {
             coord: { x: width * 0.25, y: height * 0.25 },
-            exp: convertToUnorm8(expect.topLeftColor),
+            exp: convertToUnorm8(presentColors[expect.topLeftColor]),
           },
           // Top-right.
           {
             coord: { x: width * 0.75, y: height * 0.25 },
-            exp: convertToUnorm8(expect.topRightColor),
+            exp: convertToUnorm8(presentColors[expect.topRightColor]),
           },
           // Bottom-left.
           {
             coord: { x: width * 0.25, y: height * 0.75 },
-            exp: convertToUnorm8(expect.bottomLeftColor),
+            exp: convertToUnorm8(presentColors[expect.bottomLeftColor]),
           },
           // Bottom-right.
           {
             coord: { x: width * 0.75, y: height * 0.75 },
-            exp: convertToUnorm8(expect.bottomRightColor),
+            exp: convertToUnorm8(presentColors[expect.bottomRightColor]),
           },
         ]);
       }

src/webgpu/web_platform/external_texture/video.spec.ts

@@ -18,6 +18,7 @@ import {
   kPredefinedColorSpace,
   kVideoNames,
   kVideoInfo,
+  kVideoExpectedColors,
 } from '../../web_platform/util.js';
 
 const kHeight = 16;
@@ -194,30 +195,34 @@ for several combinations of video format, video color spaces and dst color space
       passEncoder.end();
       t.device.queue.submit([commandEncoder.finish()]);
 
-      const expect = kVideoInfo[videoName].presentColors[dstColorSpace];
+      const srcColorSpace = kVideoInfo[videoName].colorSpace;
+      const presentColors = kVideoExpectedColors[srcColorSpace][dstColorSpace];
+
+      // visible rect is whole frame, no clipping.
+      const expect = kVideoInfo[videoName].display;
 
       // For validation, we sample a few pixels away from the edges to avoid compression
       // artifacts.
       t.expectSinglePixelComparisonsAreOkInTexture({ texture: colorAttachment }, [
         // Top-left.
         {
           coord: { x: kWidth * 0.25, y: kHeight * 0.25 },
-          exp: convertToUnorm8(expect.topLeftColor),
+          exp: convertToUnorm8(presentColors[expect.topLeftColor]),
         },
         // Top-right.
         {
           coord: { x: kWidth * 0.75, y: kHeight * 0.25 },
-          exp: convertToUnorm8(expect.topRightColor),
+          exp: convertToUnorm8(presentColors[expect.topRightColor]),
         },
         // Bottom-left.
         {
           coord: { x: kWidth * 0.25, y: kHeight * 0.75 },
-          exp: convertToUnorm8(expect.bottomLeftColor),
+          exp: convertToUnorm8(presentColors[expect.bottomLeftColor]),
         },
         // Bottom-right.
         {
           coord: { x: kWidth * 0.75, y: kHeight * 0.75 },
-          exp: convertToUnorm8(expect.bottomRightColor),
+          exp: convertToUnorm8(presentColors[expect.bottomRightColor]),
         },
       ]);
     });
@@ -247,16 +252,22 @@ parameters are present.
       // All tested videos are derived from an image showing yellow, red, blue or green in each
       // quadrant. In this test we crop the video to each quadrant and check that desired color
       // is sampled from each corner of the cropped image.
-      const srcVideoHeight = 240;
-      const srcVideoWidth = 320;
+      // visible rect clip applies on raw decoded frame, which defines based on video frame coded size.
+      const srcVideoHeight = source.codedHeight;
+      const srcVideoWidth = source.codedWidth;
+
+      const srcColorSpace = kVideoInfo[videoName].colorSpace;
+      const presentColors = kVideoExpectedColors[srcColorSpace][dstColorSpace];
 
-      const expect = kVideoInfo[videoName].presentColors[dstColorSpace];
+      // The test crops raw decoded videos first and then apply transform. Expectation should
+      // use coded colors as reference.
+      const expect = kVideoInfo[videoName].coded;
 
       const cropParams = [
         // Top left
         {
           subRect: { x: 0, y: 0, width: srcVideoWidth / 2, height: srcVideoHeight / 2 },
-          color: convertToUnorm8(expect.topLeftColor),
+          color: convertToUnorm8(presentColors[expect.topLeftColor]),
         },
         // Top right
         {
@@ -266,7 +277,7 @@ parameters are present.
             width: srcVideoWidth / 2,
             height: srcVideoHeight / 2,
           },
-          color: convertToUnorm8(expect.topRightColor),
+          color: convertToUnorm8(presentColors[expect.topRightColor]),
         },
         // Bottom left
         {
@@ -276,7 +287,7 @@ parameters are present.
             width: srcVideoWidth / 2,
             height: srcVideoHeight / 2,
           },
-          color: convertToUnorm8(expect.bottomLeftColor),
+          color: convertToUnorm8(presentColors[expect.bottomLeftColor]),
         },
         // Bottom right
         {
@@ -286,7 +297,7 @@ parameters are present.
             width: srcVideoWidth / 2,
             height: srcVideoHeight / 2,
           },
-          color: convertToUnorm8(expect.bottomRightColor),
+          color: convertToUnorm8(presentColors[expect.bottomRightColor]),
         },
       ];
 
@@ -381,29 +392,51 @@ compute shader, for several combinations of video format, video color spaces and
         usage: GPUTextureUsage.COPY_SRC | GPUTextureUsage.STORAGE_BINDING,
       });
 
+      // Use display size of VideoFrame and video size of HTMLVideoElement as frame size. These sizes are presenting size which
+      // apply transformation in video metadata if any.
+
       const pipeline = t.device.createComputePipeline({
         layout: 'auto',
         compute: {
-          // Shader loads 4 pixels near each corner, and then store them in a storage texture.
+          // Shader loads 4 pixels, and then store them in a storage texture.
           module: t.device.createShaderModule({
             code: `
+              override frameWidth : i32 = 0;
+              override frameHeight : i32 = 0;
               @group(0) @binding(0) var t : texture_external;
               @group(0) @binding(1) var outImage : texture_storage_2d<rgba8unorm, write>;
 
               @compute @workgroup_size(1) fn main() {
-                var yellow : vec4<f32> = textureLoad(t, vec2<i32>(80, 60));
+                let coordTopLeft = vec2<i32>(frameWidth / 4, frameHeight / 4);
+                let coordTopRight = vec2<i32>(frameWidth / 4 * 3, frameHeight / 4);
+                let coordBottomLeft = vec2<i32>(frameWidth / 4, frameHeight / 4 * 3);
+                let coordBottomRight = vec2<i32>(frameWidth / 4 * 3, frameHeight / 4 * 3);
+                var yellow : vec4<f32> = textureLoad(t, coordTopLeft);
                 textureStore(outImage, vec2<i32>(0, 0), yellow);
-                var red : vec4<f32> = textureLoad(t, vec2<i32>(240, 60));
+                var red : vec4<f32> = textureLoad(t, coordTopRight);
                 textureStore(outImage, vec2<i32>(0, 1), red);
-                var blue : vec4<f32> = textureLoad(t, vec2<i32>(80, 180));
+                var blue : vec4<f32> = textureLoad(t, coordBottomLeft);
                 textureStore(outImage, vec2<i32>(1, 0), blue);
-                var green : vec4<f32> = textureLoad(t, vec2<i32>(240, 180));
+                var green : vec4<f32> = textureLoad(t, coordBottomRight);
                 textureStore(outImage, vec2<i32>(1, 1), green);
                 return;
               }
             `,
           }),
           entryPoint: 'main',
+
+          // Use display size of VideoFrame and video size of HTMLVideoElement as frame size. These sizes are presenting size which
+          // apply transformation in video metadata if any.
+          constants: {
+            frameWidth:
+              sourceType === 'VideoFrame'
+                ? (source as VideoFrame).displayWidth
+                : (source as HTMLVideoElement).videoWidth,
+            frameHeight:
+              sourceType === 'VideoFrame'
+                ? (source as VideoFrame).displayHeight
+                : (source as HTMLVideoElement).videoHeight,
+          },
         },
       });
 
@@ -423,17 +456,21 @@ compute shader, for several combinations of video format, video color spaces and
       pass.end();
       t.device.queue.submit([encoder.finish()]);
 
-      const expect = kVideoInfo[videoName].presentColors[dstColorSpace];
+      const srcColorSpace = kVideoInfo[videoName].colorSpace;
+      const presentColors = kVideoExpectedColors[srcColorSpace][dstColorSpace];
+
+      // visible rect is whole frame, no clipping.
+      const expect = kVideoInfo[videoName].display;
 
       t.expectSinglePixelComparisonsAreOkInTexture({ texture: outputTexture }, [
         // Top-left.
-        { coord: { x: 0, y: 0 }, exp: convertToUnorm8(expect.topLeftColor) },
+        { coord: { x: 0, y: 0 }, exp: convertToUnorm8(presentColors[expect.topLeftColor]) },
         // Top-right.
-        { coord: { x: 0, y: 1 }, exp: convertToUnorm8(expect.topRightColor) },
+        { coord: { x: 0, y: 1 }, exp: convertToUnorm8(presentColors[expect.topRightColor]) },
         // Bottom-left.
-        { coord: { x: 1, y: 0 }, exp: convertToUnorm8(expect.bottomLeftColor) },
+        { coord: { x: 1, y: 0 }, exp: convertToUnorm8(presentColors[expect.bottomLeftColor]) },
         // Bottom-right.
-        { coord: { x: 1, y: 1 }, exp: convertToUnorm8(expect.bottomRightColor) },
+        { coord: { x: 1, y: 1 }, exp: convertToUnorm8(presentColors[expect.bottomRightColor]) },
       ]);
     });
   });