⚡ cache hf results in tests

[joerunde]

Description

In an effort to reduce test runtime, this uses functools.lru_cache to cache text generation results from hf transformers. This should shave about 15% off the cpu runtime based on some quick measurements on an M3.

NB: This probably will not work when running with --forked, but we don't fork the tests on github actions runs.

[github-actions]

👋 Hi! Thank you for contributing to vLLM support on Spyre.
Just a reminder: Make sure that your code passes all the linting checks, otherwise your PR won't be able to be merged. To do so, first install the linting requirements, then run format.sh and commit the changes. This can be done with uv directly:

uv sync --frozen --group lint --active --inexact

Or this can be done with pip:

uv pip compile --group lint > requirements-lint.txt
pip install -r requirements-lint.txt
bash format.sh

Now you are good to go 🚀

[jberkhahn]

Looks good to me! I'm seeing the same intermittent test failures on my PR though, not sure what they're from?

[maxdebayser]

nice!

[joerunde]

@jberkhahn the failures on all vLLM:main jobs are expected, they're just a signal that something has changed in vllm that we need to address. Catching those early like this helps us be ready when a new vllm version is released.

It does look like I missed a list -> tuple conversion in the continuous batching tests though :(

[ckadner]

Should we limit the cache size?

https://docs.python.org/3/library/functools.html#functools.lru_cache

@functools.lru_cache(maxsize=128)

Though, I don't suppose there a reason to be concerned about a growing cache?

[joerunde]

Yeah I wasn't too concerned because this should be caching relatively small objects

[joerunde]

🤔 This also doesn't appear to be reducing the runtime on github actions at all, which is a bit odd. Something seems off

[prashantgupta24]

🤔 This also doesn't appear to be reducing the runtime on github actions at all, which is a bit odd. Something seems off

I was hoping the unhashable error was the reason the caching wasn't working

[joerunde]

Ah, so the reduction in time here is actually much better when static batching and continuous batching tests are run together, since they use the same prompts and would share cache for the expected results. Separated, there are far fewer cache hits :(

Also the tests like the tkv scheduler tests that generate specific-length prompts on the fly don't benefit either. So the speedup here isn't great, but could work better if we can manage to do things like:

• Run sb and cb tests together
• Standardize the prompts we run more across tests
• Pre-fetch prompts to be used in the suite and run them through hf up-front

[joerunde]

Okay- swapping this over to use a file-based cache which should avoid loading the model w/ transformers at all in these test runs

[maxdebayser]

Shouldn't the type annotation for prompt be Union[str, list[int]]?

[maxdebayser]

Same comment about the prompt type annotation

[maxdebayser]

nice, no tokenizer required.

[maxdebayser]

LGTM. The failing main tests are due to a new upstream change fixed in #380