quantization process takes too long

[yyfcc17]

during the ptq process, it only use 1 gpu, although i have set 4 gpus in the config file (i have 4 gpus installed).

it takes almost 52 hours to quantize my 8B flux model (the Step 3 in your readme) using a L40s gpu, is this normal?

can we accelerate the ptq process by quantizing blocks parallelly on different gpus?

thanks.

[senlyu163]

Hi, i encountered OOM problem when running step 3 on a single L40s gpu. How did you solve it?

[yyfcc17]

Maybe because my flux model is smaller, only 8B, and my L40s gpu has 45GB memory, it is enough.

[senlyu163]

Maybe because my flux model is smaller, only 8B, and my L40s gpu has 45GB memory, it is enough.

Thanks for reply, i also use pixart-sigma, 800M parameters, encountered OOM on 45GB L40s gpu too. I use the original config yaml, should i modify it?

[yyfcc17]

there is a

batch_size: 256

in the config file, maybe the batch size is too large for your device.

[senlyu163]

thanks for your advice. i run sdxl-turbo on my gpus, but it costs many hours. Waiting for parallel acceleration too :)

[Chenglin-Yang]

Hi

during the ptq process, it only use 1 gpu, although i have set 4 gpus in the config file (i have 4 gpus installed).

it takes almost 52 hours to quantize my 8B flux model (the Step 3 in your readme) using a L40s gpu, is this normal?

can we accelerate the ptq process by quantizing blocks parallelly on different gpus?

thanks.

Hello @yyfcc17 , I didn't see where we can set the gpu number in the config files. May I ask how you set this?

[yyfcc17]

Hi

during the ptq process, it only use 1 gpu, although i have set 4 gpus in the config file (i have 4 gpus installed).
it takes almost 52 hours to quantize my 8B flux model (the Step 3 in your readme) using a L40s gpu, is this normal?
can we accelerate the ptq process by quantizing blocks parallelly on different gpus?
thanks.

Hello @yyfcc17 , I didn't see where we can set the gpu number in the config files. May I ask how you set this?

it's in examples/diffusion/configs/__default__.yaml, seems only for evaluation

[lmxyy]

You can adjust the number of grids used to search for the smooth factor here. However, please note that configurations other than the default value of 20 have not been tested.

[lmxyy]

I’ve created a configuration file fast_int4.yaml to accelerate the quantization process. You can replace the original int4.yaml with this file. It can reduce 50% quantization time while maintaining the same image quality on FLUX.

To further accelerate the quantization, you can adjust the num_grids in this line to 5.

[lantudou]

Hello, everyone, I just created this PR #111 to solve this problem, you can try this changes