[SB] relax constraint on min number of new tokens

[yannicks1]

[SB] relax constraint on min number of new tokens

this is relaxing an old constraint on the number of requested new tokens having to be a min of 3. Turns out it is only important that during the warmup there is at least one decode forward pass. Requesting 1 token runs prefill only during warmup -> compiler crashes (I guess it is expecting two graphs, prefill and decode) . Requesting 2+ tokens does at least one decode during warmup and thus produces a decode graph too -> things run smoothly for 2+ tokens ...

[github-actions]

👋 Hi! Thank you for contributing to vLLM support on Spyre.
Just a reminder: Make sure that your code passes all the linting checks, otherwise your PR won't be able to be merged. To do so, first install the linting requirements, then run format.sh and commit the changes. This can be done with uv directly:

uv sync --frozen --group lint --active --inexact

Or this can be done with pip:

uv pip compile --group lint > requirements-lint.txt
pip install -r requirements-lint.txt
bash format.sh

Now you are good to go 🚀

[yannicks1]

To my knowledge we previously didn't have a test for the case num_decode_tokens=3, hence I didn't add one for num_decode_tokens=2. Do we want such a test?

[sducouedic]

LGTM, thanks for fixing this

Edit 1: check code change suggestion before merging
Edit 2: feel free to add such a test, it wouldn't hurt to have one. I guess it would belong to test_spyre_warmup_shapes.py?

[joerunde]

lpgtm!

This was confusing to at least one user already who thought that meant that you also had to request at least 3 tokens in each api call, but I don't think we should focus too much on this anyway since continuous batching is almost ready

[yannicks1]

yeah, it was just something I noticed when having a look at another piece of the code..