Can the decode of transformer be accelerated by NPU

[119458]

🌱 Describe Feature Request

I trained a Transformer model. When I converted it as a whole into an mlmodel, I found that its intelligence could only be processed on the cpu. After splitting it into encode and decode, I discovered that encode could be normally accelerated using the NPU, but decode could only be processed on the cpu. Is it because decode is self-decoding, not a static issue? If decode can be accelerated by NPU, could a method for converting pt to mlmodel or mlpackage be provided
thanks

[zmkUp]

You can use Xcode to see which operators are not supported.If you're running an LLM, you can refer to an open-source library called Anemll.

[robertmsale]

For decode phase, there simply aren't large enough operations for ANE to even get a performance boost. ANE likes large, regular workloads and big tiles. Decode phase is usually very small ops over QKV & MLP. Your best bet is to take prefill phase on ANE as a win and figure out why decode phase does not prefer the GPU.