[QA] Where to download DeepSeek-R1 gptq model?

[Rane2021]

How to Download DeepSeek-v1 gptq quanted model?

[Qubitium]

You can visit https://huggingface.co/models?search=gptq to download our DeepSeek R1 distlled 7B model but we currently do not provide the full R1 model. You can use our toolkit to quantize your down R1 model.

[Rane2021]

You can visit https://huggingface.co/models?search=gptq to download our DeepSeek R1 distlled 7B model but we currently do not provide the full R1 model. You can use our toolkit to quantize your down R1 model.

Deepseek.ai has released the FP8 version. Can our tools work with it directly?
Have you considered releasing a DeepSeek R1 GPTQ quantized version? It should be very popular.

[Qubitium]

You can use the bf16 version of R1 to GPTQ quantize. We do not have large H100+ gpu to test FP8 model load. 4090 has too little vram.

https://huggingface.co/unsloth/DeepSeek-R1-BF16/tree/main

[Rane2021]

Great, Thanks!

[Rane2021]

One more question, have you tested if there are any issues with DeepSeek R1 GPTQ inference? Can it be used for inference with the vllm serve --quantization gptq method?

[Qubitium]

One more question, have you tested if there are any issues with DeepSeek R1 GPTQ inference? Can it be used for inference with the vllm serve --quantization gptq method?

There are no technical reasons why GPTQ quantized R1 cannot run on vLLM or SGLang.

[hsb1995]

@Qubitium @Rane2021

Hello, I am quite interested in your work. I would like to ask you a few questions:

1. Does this link provide the model compressed by your algorithm? https://huggingface.co/OPEA/DeepSeek-R1-int4-gptq-sym-inc
2. I saw in the demo that it supports up to 3-bit quantization. Can it be lower bit?
3. What is the difference between your work and
   https://github.com/IST-DASLab/gptq
   ? I would like to see the technical details of your paper.

[hsb1995]

You can visit https://huggingface.co/models?search=gptq to download our DeepSeek R1 distlled 7B model but we currently do not provide the full R1 model. You can use our toolkit to quantize your down R1 model.

Could you please tell me which deepseek-7B model you can compress? If convenient, please provide the link of 7B model.

[Qubitium]

@hsb1995

1. The link you referred to is a GPTQ quant model made by AutoRound. However, that model has not been benchmarked, that i am aware of so I can't say one or the other how good it is. AutoRound does not use the same algorithm but generated the a model format that is compatible with GPTQ.
2. Please check https://github.com/ModelCloud/GPTQModel#citation for link to the papers. We use the same original GPTQ alogorithm pioneered by IST-DASLab.
3. Please check our readme for link to our quantized DeepSeek 7B model with full-benchmarks. https://github.com/ModelCloud/GPTQModel#quality-gptq-4bit-50-bpw-can-match-bf16

[hsb1995]

https://arxiv.org/abs/2210.17323
Hello professor, is this paper your project's paper?

[hsb1995]

https://arxiv.org/abs/2210.17323 Hello professor, is this paper your project's paper?

@Qubitium

[Qubitium]

https://arxiv.org/abs/2210.17323 Hello professor, is this paper your project's paper?

This paper was written by the original researchers of GPTQ. GPTQModel is code, based on the original code from the original research team plus many modifications on usage, inference, and quantization.