Skip to content

How to transform the orignal Mixtral 8*7B into the mixed HQQ quantized model ? #40

@eljrte

Description

@eljrte

Really appreciate your great work. As it helps me to run MoE on a consumer GPU.
I wonder how u transform the original Mixtral 8*7B into the quantized one using HQQ , as I found your model.safetensors.index.json very special , each part has its own safetensors.
Do u have any script or can u tell me the way briefly?
I appreciate it very much.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions