Can't using llmcompressor for awq quantization and using autoawq completely replace each other?

[dingjingzhen]

The models quantized using llmcompressor for awq and those quantized using autoawq are different, including shape, naming, etc. I hope the structures of the two can be the same. Can this be achieved

[brian-dellabetta]

Hi @dingjingzhen , our AWQ implementation is very similar to AutoAWQ, but the model checkpoints will probably not align in terms of shapes and naming. We use a "pack-quantized" approach for 4-bit weights, packing 8 of them into a single 32-bit. I am not sure how AutoAWQ handles this. They are loaded up in vllm using separate pathways as well.

All that said, they should work functionally in the same way on vllm. Do you see a drop in performance or accuracy between the two? What is the use case for having the structure be the exact same? We will be revisiting AWQ in the next month, updating with some performance improvements and better defaults.

[Leoooooo123]

@brian-dellabetta what is the order when you pack 8 4-bit weights into one single 32-bit? in the implementation of autoawq, they have what they call order_map, the implementation detail is as follows:

def gemm_pack(self, weight, scales, zeros, quant_config):
        if zeros is not None:
            zeros = zeros.t().contiguous()
        scales = scales.t().contiguous()
        weight = weight.to(torch.int32).t().contiguous()

        bit = quant_config["weight"]["bit"]
        pack_num = 32 // bit

        int_weight = torch.zeros(
            (weight.shape[0], weight.shape[1] // 32 * bit),
            dtype=torch.int32,
            device=weight.device,
        )

        for col in range(weight.shape[1] // pack_num):
            if bit == 4:
                order_map = [0, 2, 4, 6, 1, 3, 5, 7]
            else:
                raise NotImplementedError("Only 4-bit are supported for now.")
            for i in range(pack_num):
                int_weight_col = weight[:, col * pack_num + order_map[i]]
                int_weight[:, col] |= int_weight_col << (i * bit)

        if zeros is not None:
            int_zeros = torch.zeros(
                (zeros.shape[0], zeros.shape[1] // 32 * bit),
                dtype=torch.int32,
                device=zeros.device,
            )

            for col in range(zeros.shape[1] // pack_num):
                if bit == 4:
                    order_map = [0, 2, 4, 6, 1, 3, 5, 7]
                else:
                    raise NotImplementedError("Only 4-bit are supported for now.")
                for i in range(pack_num):
                    intzero_col = zeros[:, col * pack_num + order_map[i]]
                    int_zeros[:, col] |= intzero_col << (i * bit)
        else:
            int_zeros = None
        del weight
        return int_weight, scales, int_zeros

[brian-dellabetta]

Hi @Leoooooo123 , our implementation is the PackQuantizedCompressor here, implementation for packing here.

I don't think think we do any ordering along the lines of the order_map AutoAWQ is using, just grab the next sequential set of 8 4-bit ints. Do you know why they do that in AutoAWQ?

[brian-dellabetta]

FYI we've opened an issue to add a converter from AutoAWQ checkpoint format to compressed-tensors

• [Feature Request][Help Wanted] Convert AutoAWQ checkpoints to compressed-tensors #1962

You can track progress there, we are hoping a community user would be so kind to add. I will post the order_map detail there