Issue with 50-Series Graphics Card Compatibility for xFormers

[KATe-AGI]

🐛 Bug

Hello, my environment is equipped with PyTorch 2.8 and CUDA 12.8. When I run pip3 install -U xformers --index-url https://download.pytorch.org/whl/cu128 with my 50-series graphics card, xFormers can be compiled and installed successfully. However, xFormers does not work during actual training—the error message indicates that the issue is kernel architecture incompatibility. I suspect if this precompiled package is only intended for H-series graphics cards?

[sirus20x6]

also have this issue on rtx pro 6000

[daitisuki]

you can try this guy's idea I found,I solve this problem by it:
It's because the build check in xformers incorrectly enables FA3 for Blackwell, while FA3 only supports GA100/sm80 and Hopper/sm90.

The invalid argument is triggered by setting an unsupported shared memory size, I think due to limited shared memory size in customer cards (RTX Blackwell/sm120).

I've found a PR #1262 resolving this issue, which hasn't been merged yet.

For a temporary workaround, I changed line 19 in xformers/ops/fmha/dispatch.py, setting _USE_FLASH_ATTENTION_3 to False (changing the file in the site-packages directory is enough, no need to rebuild).

[guidenglong1984]

you can try this guy's idea I found,I solve this problem by it: It's because the build check in xformers incorrectly enables FA3 for Blackwell, while FA3 only supports GA100/sm80 and Hopper/sm90.

The invalid argument is triggered by setting an unsupported shared memory size, I think due to limited shared memory size in customer cards (RTX Blackwell/sm120).

I've found a PR #1262 resolving this issue, which hasn't been merged yet.

For a temporary workaround, I changed line 19 in xformers/ops/fmha/dispatch.py, setting _USE_FLASH_ATTENTION_3 to False (changing the file in the site-packages directory is enough, no need to rebuild).

thanks，niubi！working on xformers 0.0.32 post2(torch2.8+cuda12.9+python3.12)