🐛 implement better checking for granite

[joerunde]

Fix granite-3.3-8b detection

Fixes #498

This PR updates the logic that checks for granite-3.3-8b-instruct. Instead of matching on the model name, it checks the hf config type and checks for specific values related to the 8b granite variant.

This also consolidates all of the granite-specific overrides to one method inside platform.py. This should make it much easier to both:

• Update these overrides for granite 3.3 8b
• Add overrides for other models

in the future.

The number of blocks override for the spyre kv cache size was particularly interesting. We were in a halfway state where we took overriddes from --num-gpu-blocks-override and saved them off to platform.py and then used the override to change the scheduler's behavior. I've updated things so that the scheduler no longer needs that extra override so that we can simply use the configured override everywhere. This is more robust since the config value is serialized and given to subprocesses, and more maintainable than managing two separate numbers for this purpose.

[github-actions]

👋 Hi! Thank you for contributing to vLLM support on Spyre.
Just a reminder: Make sure that your code passes all the linting checks, otherwise your PR won't be able to be merged. To do so, first install the linting requirements, then run format.sh and commit the changes. This can be done with uv directly:

uv sync --frozen --group lint --active --inexact

Or this can be done with pip:

uv pip compile --group lint > requirements-lint.txt
pip install -r requirements-lint.txt
bash format.sh

Now you are good to go 🚀

[prashantgupta24]

I wonder if we have to account for other granite models down the road

[joerunde]

Very likely, given current timeframes for model support

[joerunde]

bot:test
MARKERS="spyre and cb and not quantized"

[joerunde]

bot:test
MARKERS="spyre and cb and not quantized"

[joerunde]

bot:test
MARKERS="spyre and cb and not quantized"

[joerunde]

@gkumbhat @tjohnson31415 I verified this with:

export VLLM_SPYRE_USE_CB=1

# Boot up with granite 33 8b
vllm serve ibm-granite/granite-3.3-8b-instruct --max-model-len 256 --max-num-seqs 4 --tens 4

# Symlink model to somewhere else
ln -s /models/huggingface_cache/hub/models--ibm-granite--granite-3.3-8b-instruct/snapshots/51dd4bc2ade4059a6bd87649d68aa11e4fb2529b/ /tmp/granite

# Disable compilation and boot again from the symlink
DISABLE_COMPILATION=1 vllm serve /tmp/granite --max-model-len 256 --max-num-seqs 4 --tens 4

Seems to work! Logs are as expected as well, showing

Spyre KV cache size: 133120 tokens

which is 2080 * 64

[gkumbhat]

Looks good to me!