[Bug]: documentation around multiple modifiers

[koush]

⚙️ Your current environment

N/A

🐛 Describe the bug

I'm trying to use the example for multiple modifiers to quantize qwen3 moe with both awq and fp8, but seem to be running into an error when performing inference (Linear key is missing). I'm not sure if vllm actually supports this yet.

https://github.com/vllm-project/llm-compressor/blob/main/examples/quantization_non_uniform/quantization_multiple_modifiers.py

To try to isolate the issue, I am trying to perform AWQ on only specific layers. However, this results in vllm failing with the Linear key missing.

(EngineCore_DP0 pid=196393) ERROR 10-13 19:28:10 [core.py:708]     config = self.quant_config.target_scheme_map["Linear"].get("weights")

Here is the recipe:

recipe = [
    AWQModifier(
        duo_scaling=False,
        scheme="W4A16",
        targets=[
            "re:.*down_proj",
            "re:.*up_proj",
            "re:.*gate_proj",
        ],
        mappings=[
            AWQMapping(
                r"re:.*?\.1\..*?up_proj",
                ["re:.*down_proj$"],
            ),
        ],
    ),
]

Does llm compressor and/or vllm currently support this?

🛠️ Steps to reproduce

No response

[brian-dellabetta]

Hi @koush , your recipe mappings look a little off. MoE models have a slightly different mapping -- https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/modifiers/awq/mappings.py#L42-L56

Try with those instead, if you want to focus on just mlp layers you can do

recipe = [
    AWQModifier(
        duo_scaling=False,
        scheme="W4A16",
        targets=[
            "re:.*down_proj$",
            "re:.*up_proj$",
            "re:.*gate_proj$",
        ],
        mappings=[
    AWQMapping(
        "re:.*post_attention_layernorm$",
        ["re:.*mlp.experts.*.gate_proj$", "re:.*mlp.experts.*.up_proj$"],
    ),
    AWQMapping(
        "re:.*up_proj$",
        ["re:.*down_proj$"],
    ),
        ],
    ),
]

vllm should support different quantization schemes applied to different layers of the same model. If the error persists please provide a full stack trace that we can inspect

[koush]

Thanks @brian-dellabetta , I have seen those defaults and they actually served as a starting point for attempting to target by layer.

There's a couple issues:

The name matching does not take into account the parent module (which makes targeting by layer impossible?)

llm-compressor/src/llmcompressor/modifiers/awq/base.py

Line 329 in 6c07ef3

for balance_suffix, balance_layer in match_named_modules(

And then the matching in vllm does not account for 2 regex:

https://github.com/vllm-project/vllm/blob/314285d4f2134111345b4c4cbb583a925db9a91d/vllm/model_executor/layers/quantization/compressed_tensors/utils.py#L162

If this is possible to do now, could you share a recipe?

[brian-dellabetta]

If you want the AWQ smoothing algorithm to only run on a certain layer, you can do (e.g. for layer 10):

    AWQMapping(
        r"re:.*layers\.10.*post_attention_layernorm$",
        [r"re:.*layers\.10.*mlp.experts.*.gate_proj$", r"re:.*layers\.10.*mlp.experts.*.up_proj$"],
    ),
    AWQMapping(
        r"re:.*layers\.10.*up_proj$",
        [r"re:.*layers\.10.*down_proj$"],
    ),

But if you are trying to track down why this won't load in vllm, you can just use QuantizationModifier to quantize the specific layer that won't load up.

Start with a basic QuantizationModifier example in our examples/. If that won't load in vllm, there's no point in trying something more complex like multi-modifiers

[koush]

@brian-dellabetta thanks, I will give this a shot. What should the targets be in this instance?

[koush]

Confirmed that this mapping does not work, because in the aforementioned code from my comment earlier, the name match is not prefixed with the parent module.

For example, AWQMapping attempts to match

r"re:.*layers\.10.*down_proj$"

But the names being matched are without the parent layer, and are in the layer prefix stripped form:

mlp.experts.17.down_proj

So AWQ matching returns an empty set, and nothing is quantized resulting in a full bf16 model being output.

[koush]

Attempting to use

recipe = QuantizationModifier(targets="Linear", scheme="NVFP4A16", ignore=["lm_head", "re:.*mlp.gate$"])

results in:

(EngineCore_DP0 pid=266281) Process EngineCore_DP0:
(EngineCore_DP0 pid=266281) Traceback (most recent call last):
(EngineCore_DP0 pid=266281)   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=266281)     self.run()
(EngineCore_DP0 pid=266281)   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=266281)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=266281)   File "/root/chat/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 712, in run_engine_core
(EngineCore_DP0 pid=266281)     raise e
(EngineCore_DP0 pid=266281)   File "/root/chat/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 699, in run_engine_core
(EngineCore_DP0 pid=266281)     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=266281)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=266281)   File "/root/chat/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 498, in __init__
(EngineCore_DP0 pid=266281)     super().__init__(vllm_config, executor_class, log_stats,
(EngineCore_DP0 pid=266281)   File "/root/chat/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 83, in __init__
(EngineCore_DP0 pid=266281)     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=266281)                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=266281)   File "/root/chat/.venv/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__
(EngineCore_DP0 pid=266281)     self._init_executor()
(EngineCore_DP0 pid=266281)   File "/root/chat/.venv/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 55, in _init_executor
(EngineCore_DP0 pid=266281)     self.collective_rpc("load_model")
(EngineCore_DP0 pid=266281)   File "/root/chat/.venv/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc
(EngineCore_DP0 pid=266281)     return [run_method(self.driver_worker, method, args, kwargs)]
(EngineCore_DP0 pid=266281)             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=266281)   File "/root/chat/.venv/lib/python3.12/site-packages/vllm/utils/__init__.py", line 3122, in run_method
(EngineCore_DP0 pid=266281)     return func(*args, **kwargs)
(EngineCore_DP0 pid=266281)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=266281)   File "/root/chat/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 213, in load_model
(EngineCore_DP0 pid=266281)     self.model_runner.load_model(eep_scale_up=eep_scale_up)
(EngineCore_DP0 pid=266281)   File "/root/chat/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 2635, in load_model
(EngineCore_DP0 pid=266281)     self.model = model_loader.load_model(
(EngineCore_DP0 pid=266281)                  ^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=266281)   File "/root/chat/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/base_loader.py", line 45, in load_model
(EngineCore_DP0 pid=266281)     model = initialize_model(vllm_config=vllm_config,
(EngineCore_DP0 pid=266281)             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=266281)   File "/root/chat/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/utils.py", line 63, in initialize_model
(EngineCore_DP0 pid=266281)     return model_class(vllm_config=vllm_config, prefix=prefix)
(EngineCore_DP0 pid=266281)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=266281)   File "/root/chat/.venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_moe.py", line 593, in __init__
(EngineCore_DP0 pid=266281)     self.model = Qwen3MoeModel(vllm_config=vllm_config,
(EngineCore_DP0 pid=266281)                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=266281)   File "/root/chat/.venv/lib/python3.12/site-packages/vllm/compilation/decorators.py", line 201, in __init__
(EngineCore_DP0 pid=266281)     old_init(self, vllm_config=vllm_config, prefix=prefix, **kwargs)
(EngineCore_DP0 pid=266281)   File "/root/chat/.venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_moe.py", line 389, in __init__
(EngineCore_DP0 pid=266281)     self.start_layer, self.end_layer, self.layers = make_layers(
(EngineCore_DP0 pid=266281)                                                     ^^^^^^^^^^^^
(EngineCore_DP0 pid=266281)   File "/root/chat/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 630, in make_layers
(EngineCore_DP0 pid=266281)     maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}"))
(EngineCore_DP0 pid=266281)                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=266281)   File "/root/chat/.venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_moe.py", line 391, in <lambda>
(EngineCore_DP0 pid=266281)     lambda prefix: Qwen3MoeDecoderLayer(vllm_config=vllm_config,
(EngineCore_DP0 pid=266281)                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=266281)   File "/root/chat/.venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_moe.py", line 331, in __init__
(EngineCore_DP0 pid=266281)     self.mlp = Qwen3MoeSparseMoeBlock(vllm_config=vllm_config,
(EngineCore_DP0 pid=266281)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=266281)   File "/root/chat/.venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_moe.py", line 144, in __init__
(EngineCore_DP0 pid=266281)     self.experts = FusedMoE(num_experts=self.n_routed_experts,
(EngineCore_DP0 pid=266281)                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=266281)   File "/root/chat/.venv/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/layer.py", line 1104, in __init__
(EngineCore_DP0 pid=266281)     else quant_config.get_quant_method(self, prefix))
(EngineCore_DP0 pid=266281)          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=266281)   File "/root/chat/.venv/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors.py", line 139, in get_quant_method
(EngineCore_DP0 pid=266281)     return CompressedTensorsMoEMethod.get_moe_method(self, layer)
(EngineCore_DP0 pid=266281)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=266281)   File "/root/chat/.venv/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe.py", line 140, in get_moe_method
(EngineCore_DP0 pid=266281)     elif quant_config._is_dynamic_token_w8a8(weight_quant, input_quant):
(EngineCore_DP0 pid=266281)          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=266281)   File "/root/chat/.venv/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors.py", line 322, in _is_dynamic_token_w8a8
(EngineCore_DP0 pid=266281)     is_8_bits = weight_quant.num_bits == input_quant.num_bits == 8
(EngineCore_DP0 pid=266281)                                          ^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=266281) AttributeError: 'NoneType' object has no attribute 'num_bits'

[brian-dellabetta]

NVFP4A16 only works with Nvidia Blackwell GPUs or higher. Try

recipe = QuantizationModifier(targets="Linear", scheme="W4A16", ignore=["lm_head", "re:.*mlp.gate$"])

[koush]

NVFP4A16 only works with Nvidia Blackwell GPUs or higher. Try

recipe = QuantizationModifier(targets="Linear", scheme="W4A16", ignore=["lm_head", "re:.*mlp.gate$"])

For what it's worth I am on an RTX Pro 6000 with cuda 12.9, but I will give this a shot as well thanks.