Whether DSA of prefill stage use MQA mode or MHA mode?

[oliverYoung2001]

The code of inference demo (https://github.com/deepseek-ai/DeepSeek-V3.2-Exp/blob/main/inference/model.py#L563) shows that DSA of prefill stage uses MHA mode. However, in the technique report, the author claims that DSA is instantiated based on MQA mode. And in FlashMLA repo, the DSA for prefill stage only supports MQA mode. Is it a bug in the inference demo?

[EricLina]

Same question, it seems that prefilling couldn't be accelerated by MQA.

[EricLina]

There is no backward kernel code for sparse MQA in FlashMLA repo, but in the report they said it is trained with sparse kernel in the second stage, idk would Deepseek team release this part of code.

[EricLina]

@youkaichao Could you take a look?