Add: targets and ignore inference for sparse compression

[rahul-tuli]

This PR adds support to automatically infer targets and ignore lists for sparse compression

The lists thus generated will be used to to ignore things like layernorms, and embeddings

The logic for adding to targets:

• layer must have sparsity > threshold (set to 20% for now)
• layer must follow the sparsity structure defined

Otherwise it's added to the ignore list

Note: We also perform a reduction of the targets, and ignore list to make them cleaner and easily readable inside config.json

The compression_config inside config.json will now look like:

"compression_config": {
    "sparsity_config": {
      "format": "sparse-bitmask",
      "global_sparsity": 0.12233772669884009,
      "ignore": [
        "Embedding",
        "lm_head",
        "LlamaRotaryEmbedding",
        "SiLU",
        "LlamaRMSNorm"
      ],
      "registry_requires_subclass": false,
      "sparsity_structure": "2:4",
      "targets": [
        "Linear"
      ]
    }
  },

After review comments we were able to further simplify compression_config, now we only include modules in ignore list if they are targetted, now the compression config looks like:

  "compression_config": {
    "sparsity_config": {
      "format": "dense",
      "global_sparsity": 0.12233772669884009,
      "ignore": [
        "lm_head"
      ],
      "registry_requires_subclass": false,
      "sparsity_structure": "2:4",
      "targets": [
        "Linear"
      ]
    }
  }

Note: relies on neuralmagic/compressed-tensors#159

[github-actions]

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

[kylesayrs]

Looks good! I'd prefer to call "layers" modules, since layers is already used to describe transformer blocks, but I think readers still understand.

[rahul-tuli]

Looks good! I'd prefer to call "layers" modules, since layers is already used to describe transformer blocks, but I think readers still understand.

Let's be consistent and use module/submodule over layer wherever possible

[mgoin]

Nice LGTM

[dsikka]

@rahul-tuli The Sparsification tests are failing with a different error than the shared memory issue we've been seeing. Can you look into it?

[rahul-tuli]

@rahul-tuli The Sparsification tests are failing with a different error than the shared memory issue we've been seeing. Can you look into it?

Fixed!