Quantized model time performance

[HemanLin-cl]

Dear authors,

Thanks for the great help. Currently we've tried to measure SDXL base model.
I've tried (win-vulkan)

sd-cli.exe -M convert -m sd_xl_diffusion_base_x1.safetensors -o sd_xl_diffusion_base_x1_q8_0.gguf -v --type q8_0
sd-cli.exe -M convert -m sd_xl_diffusion_base_x1.safetensors -o sd_xl_diffusion_base_x1_q4_0.gguf -v --type q4_0

I can convert SDXL into q8_0/q4_0 format by applying this.
And then I tried to compare fp16/q8_0/q4_0 models' time by:

sd-cli.exe -m sd_xl_diffusion_base_x1.safetensors -W 1024 -H 1024 -p "a lovely cat" --vae-tiling --vae-conv-direct --steps 7 -v
sd-cli.exe -m sd_xl_diffusion_base_x1_q8_0.gguf -W 1024 -H 1024 -p "a lovely cat" --vae-tiling --vae-conv-direct --type q8_0 -v --steps 7
sd-cli.exe -m sd_xl_diffusion_base_x1_q4_0.gguf -W 1024 -H 1024 -p "a lovely cat" --vae-tiling --vae-conv-direct --type q4_0 -v --steps 7

However, the time performances of the three models are almost the same.
FP16: 64 sec, q8_0: 68 sec, q4_0: 63 sec
Is this expected?

My platform info is as following:

Lunar Lake
CPU: Intel Ultra 268V
GPU: Intel R AI Boost
RAM: 32G

Thanks

[stduhpf]

Yes, I'd say this is expected.

Fp16 is a natively supported type on your hardware, Q8_0 and Q4_0 are not.

When a type is not natively supported, there's some compute overhead because most operations need to either dequantize the weights to a native type or use some software tricks to compute the results directly with the quantized type, which may be faster than dequantizing before the native OP, but still slower than the native OP alone.

Diffusion models are mostly bottlenecked by the compute power, reducing the size of the model doesn't help that much, unlike with LLMs which are mostly bottlenecked by the memory bandwidth. It still helps a bit, which might explain why the Q4_0 can take the edge over fp16.

[bssrdf]

There is one more fundamental factor: the intermediate tensor output is forced to be FP32 (mainly GEMM, FA and NORM etc.) no matter what types the weights are. This is limiting the performance. See #1104 which got a speed up by using the full FP16/BF16 pipeline (it targets CUDA backend though).