Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Execution tests for subgroupAny and subgroupAll #3924

Merged
merged 4 commits into from
Aug 30, 2024

Conversation

alan-baker
Copy link
Contributor

  • Compute tests with all active invocation and partially active invocations
  • Fragment tests with all active invocations
  • Removed unimplemented dynamically uniform subgroupBroadcast test (due to const requirement)

Issue: #


Requirements for PR author:

  • All missing test coverage is tracked with "TODO" or .unimplemented().
  • New helpers are /** documented */ and new helper files are found in helper_index.txt.
  • Test behaves as expected in a WebGPU implementation. (If not passing, explain above.)
  • Test have be tested with compatibility mode validation enabled and behave as expected. (If not passing, explain above.)

Requirements for reviewer sign-off:

  • Tests are properly located in the test tree.
  • Test descriptions allow a reader to "read only the test plans and evaluate coverage completeness", and accurately reflect the test code.
  • Tests provide complete coverage (including validation control cases). Missing coverage MUST be covered by TODOs.
  • Helpers and types promote readability and maintainability.

When landing this PR, be sure to make any necessary issue status updates.

@alan-baker alan-baker requested a review from dneto0 August 30, 2024 14:47
Copy link
Contributor

@dneto0 dneto0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This goes a long way, thanks!

Are you planning to write fragment tests with split subgroups? If so, then add an .unimplemented() test for it.

- got: ${res}`);
}
} else {
if (res !== 999) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This 999 is mysterious. I see it's the default value in the output buffer, as set by runComputeTest.

The 999 should be a defined constant exported from subgroup_util.ts.

}`;

const prng = new PRNG(t.params.case);
// Case 0 is all 0s.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, thanks for putting the two special cases in. It's important!

const prng = new PRNG(t.params.case);
// Case 0 is all 0s.
// Case 1 is all 1s.
// Other cases are filled with random 0s and 1s.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's good to have a few random ones.

But this allows an incorrect but simple implementation to nearly always pass this test: Look at the first half of the bits only.

How about:
increase kNumCases to 15
a decl just before inputData to pick out which bit to set to 0.
let dropout = prng.uniformInt(32);
/// Cases 2 to 9 are many 1 bits except for a single randomly selected 0 (in position 0..31)
/// Other cases are filled with random 0s and 1s.
then inside the iterRange:
...
else if (t.params.case < 10) {
return x === dropout ? 0 : 1;
} else {
return prng.uniformInt(2);
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I recommend making a helper function to generate the input data, since the same logic is used twice and is (going to get) pretty complicated.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea. I think this has expoed a problem with the fragment tests.

@builtin(position) pos : vec4f,
) -> @location(0) vec2u {
var subgroup_id = 0u;
if subgroupElect() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I recommend avoiding divergence here.
The subgroup representative IDs don't have to be sequential. So it's ok to delete the if condition; it's ok for every invocation to bump the counter. The broadcast on line 372 still sends the representative ID to all the invocations in the subgroup.

@alan-baker
Copy link
Contributor Author

This goes a long way, thanks!

Are you planning to write fragment tests with split subgroups? If so, then add an .unimplemented() test for it.

I've tried a little, but it gets tricky quickly. Compilers collapsing conditional code can easily lead to too many active invocations as was shown in the experimental reconvergence tests. So for now I'm not planning on adding those tests and relying on the compute versions.

@dneto0
Copy link
Contributor

dneto0 commented Aug 30, 2024

I've tried a little, but it gets tricky quickly.

ok, sounds good. Let's make sure this is tracked somehow: an .unimplemented() test seems best so it has a chance of getting revisited in the future.

Copy link
Contributor Author

@alan-baker alan-baker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fragment tests are trickier to avoid problematic compiler optimizations which make generating expectations unreliable so I'm not planning to introduce them now.

There does seem to be a problem with the fragment variants of the test though.

const prng = new PRNG(t.params.case);
// Case 0 is all 0s.
// Case 1 is all 1s.
// Other cases are filled with random 0s and 1s.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea. I think this has expoed a problem with the fragment tests.

@alan-baker
Copy link
Contributor Author

Decided to pull the fragment tests for now.

@alan-baker alan-baker requested a review from dneto0 August 30, 2024 18:45
Copy link
Contributor

@dneto0 dneto0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

* Compute tests with all active invocation and partially active
  invocations
* Fragment tests with all active invocations
* Removed unimplemented dynamically uniform subgroupBroadcast test (due
  to const requirement)
* Remove fragment tests for now
* Add helpers to generate input data and add new class of cases
* Increase number of cases to 15 per variants
* Export data sentinel value
@alan-baker alan-baker enabled auto-merge (squash) August 30, 2024 18:56
@alan-baker alan-baker merged commit 9b30f7a into gpuweb:main Aug 30, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: No status
Development

Successfully merging this pull request may close these issues.

2 participants