`wgpu` injects compute dispatches and prevents reuse of pipelines, leading to massive performance issues

[djeedai]

Description

wgpu does indirect args validation on Vulkan at least by letting the user code bind pipelines and groups, then overwrite all of that with its own injected pipeline and compute dispatch, then auto-rebind the user's pipeline and finally execute the user's compute dispatch. This doubles the number of pipeline switch, and completely defeats any batching by pipeline which was precisely intended to reduce the heavy cost of switching pipeline.

The InstanceFlag::VALIDATION_INDIRECT_CALL added in v26 to presumably disable this as I understand is unusable since it breaks several DX12 features.

Repro steps

Expected vs observed behavior

wgpu doesn't auto-inject any command that the user didn't explicitly perform, especially not for validation.

Platform

Windows 11 + Vulkan, this has been on-going for at least 3-4 versions, probably more. Currently at v26.

[ErichDonGubler]

Would resolving #8255 also resolve this issue?

[djeedai]

@ErichDonGubler yes it would. I assume they're the same issue phrased in different ways. I'm guessing if those things (validation and DX12 features) are under the same flag it's because they're linked in the implementation, not by design choice, that'd be odd.

[inner-daemons]

The InstanceFlag::VALIDATION_INDIRECT_CALL added in v26 to presumably disable this as I understand is unusable since it breaks several DX12 features.

What did you mean by this? I also don't see why this would be addressed by #8255

[djeedai]

https://docs.rs/wgpu/latest/wgpu/struct.InstanceFlags.html#associatedconstant.VALIDATION_INDIRECT_CALL

Disabling this will also cause the following built-ins to not report the right values on the D3D12 backend:

• the 3 components of @Builtin(num_workgroups) will be 0
• the value of @Builtin(vertex_index) will not take into account the value of the first_vertex/base_vertex argument present in the indirect buffer
• the value of @Builtin(instance_index) will not take into account the value of the first_instance argument present in the indirect buffer

Disabling the validation via this flag suddenly breaks several unrelated features of WGSL, and only on the DX12 backend. Anyone using those features and either targeting DX12 or shipping a multi-platform library therefore cannot use this flag, so cannot disable validation.

I also don't see why this would be addressed by #8255

#8255 asks to split the validation enable/disable from the DX12 features being disabled/broken. My guess is that there was never an actual design intent to break those DX12 features while disabling that flag, but that probably the DX12 backend implementation relies on the validation pass to also do some work which makes those features available. And therefore by disabling validation, this breaks the features. This is only a guess, but I don't see why one would ship a flag which at the same time disable an optional validation not present in the WebGPU/WGSL spec. AND break some feature on a single backend. Unless this is as I suspect a stop-gap solution until the DX12 backend properly decouples these. Just some guesses.

In any case, if those are splits, then we can disable validation and mitigate the performance loss, while retaining the usage of those DX12 features. Which is why @pcwalton logged #8255, which is an alternative solution to the same underlying performance issue we both have from the same project (Patrick is a contributor on the project I maintain).

[cwfitzgerald]

To add some information here, another reason it would be useful to separate these is that we can actually eliminate the compute shader entirely and still have functionality work. It would unfortunately require having a different pipeline internally for indirect and direct draw calls but this may be worth it? I will file an issue about this.

• num_workgroups we can emulate as-is by binding the indirect buffer as an SRV storage buffer bound as a root descriptor so the shader can read its own indirect buffer with the information.
• vertex_index/instance_index can use this same trick.
• on SM6.8, vertex_index/instance_index can use the new SV_StartVertexLocation and SV_StartInstanceLocation.

The only thing we're not able to emulate would be a gl_DrawID which we don't support yet in wgsl, but is part of D3D12's "ExecuteIndirectTier 1.1" which unfortunately only nvidia and qualcomm support currently.

wgpu does indirect args validation on Vulkan at least by letting the user code bind pipelines and groups, then overwrite all of that with its own injected pipeline and compute dispatch, then auto-rebind the user's pipeline and finally execute the user's compute dispatch. This doubles the number of pipeline switch, and completely defeats any batching by pipeline which was precisely intended to reduce the heavy cost of switching pipeline.

This just sounds like a straight bug, we shouldn't be binding the user's pipeline twice.

[pcwalton]

We don't use gl_DrawID anywhere. Being able to get rid of the compute shader dispatch entirely by simply binding the indirect buffer in the shader would be huge.