Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reftests: Allow large totalPixels for fuzzy matching #3230

Merged
merged 3 commits into from
Dec 21, 2023

Conversation

kainino0x
Copy link
Collaborator

@kainino0x kainino0x commented Dec 21, 2023

For these tests the fuzzy matching is only to allow for slight differences like rounding errors. We don't care about how many pixels are different.
https://web-platform-tests.org/writing-tests/reftests.html#fuzzy-matching

Also adds fuzzy matching to two more canvas_colorspace tests because it's not important to have extremely precise results for any test that does color conversions or blending (at least right now).

I haven't tested this exact change but in the linked Chromium CL below we saw the following:

01:01:43.047 7604 Found pixels_different: 500, max_channel_diff: 1
Allowed pixels_different; 0-400, max_channel_diff: 0-2
01:01:44.062 7604 Found pixels_different: 700, max_channel_diff: 2
Allowed pixels_different; 0-400, max_channel_diff: 0-2

So this should definitely work.

Some other tests that we're not currently passing for other reasons render quite large areas, so we set totalPixels very large. For example the canvas_colorspace tests render twelve (128 * devicePixelRatio)-squared canvases.

Issue: #1045, https://chromium-review.googlesource.com/c/chromium/src/+/5134470


Requirements for PR author:

  • All missing test coverage is tracked with "TODO" or .unimplemented().
  • New helpers are /** documented */ and new helper files are found in helper_index.txt.
  • Test behaves as expected in a WebGPU implementation. (If not passing, explain above.)

Requirements for reviewer sign-off:

  • Tests are properly located in the test tree.
  • Test descriptions allow a reader to "read only the test plans and evaluate coverage completeness", and accurately reflect the test code.
  • Tests provide complete coverage (including validation control cases). Missing coverage MUST be covered by TODOs.
  • Helpers and types promote readability and maintainability.

When landing this PR, be sure to make any necessary issue status updates.

For these tests the fuzzy matching is only to allow for slight
differences like rounding errors. We don't care about how many pixels
are different.

Also adds fuzzy matching to two more canvas_colorspace tests because
it's not important to have extremely precise results for any test that
does color conversions or blending (at least right now).

Issue: gpuweb#1045, https://chromium-review.googlesource.com/c/chromium/src/+/5134470
@kainino0x kainino0x requested a review from austinEng December 21, 2023 21:59
@kainino0x kainino0x removed the request for review from austinEng December 21, 2023 22:11
@greggman greggman self-requested a review December 21, 2023 22:17
Copy link
Contributor

@greggman greggman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

I find it strange that totalPixels is any difference even if they are under maxDifference but 🤷‍♂️

@kainino0x
Copy link
Collaborator Author

Yeah, it's unintuitive, but that's how it's defined. I think they wanted to define it such that you could use "fuzzy" matching to almost-precisely match a particular expected error. But I don't know if it actually is ever used like that.
https://web-platform-tests.org/writing-tests/reftests.html#fuzzy-matching

@kainino0x kainino0x enabled auto-merge (squash) December 21, 2023 22:24
@kainino0x kainino0x disabled auto-merge December 21, 2023 22:25
@kainino0x kainino0x enabled auto-merge (squash) December 21, 2023 22:25
@kainino0x kainino0x merged commit fdd7367 into gpuweb:main Dec 21, 2023
1 check passed
@kainino0x kainino0x deleted the morefuzzy branch December 21, 2023 22:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants