[CB] Test continuous batching through the scheduler

[sducouedic]

• Closes [CB] Test corner cases #167 , [CB] Test stripping of repeated left padding #166, [CB] Test scheduling constraints #165

[github-actions]

👋 Hi! Thank you for contributing to vLLM support on Spyre.
Just a reminder: Make sure that your code passes all the linting checks, otherwise your PR won't be able to be merged. To do so, first install the linting requirements, then run format.sh and commit the changes. This can be done with uv directly:

uv sync --frozen --group lint --active --inexact

Or this can be done with pip:

uv pip compile --group lint > requirements-lint.txt
pip install -r requirements-lint.txt
bash format.sh

Now you are good to go 🚀

[prashantgupta24]

I like it!

[prashantgupta24]

From what I understand tkv is not 64 because this is not a new batch yet since request 1 is still running?

[sducouedic]

you're other comment is correct, it is about calling reduce_left_padding

[prashantgupta24]

The left padding is not removed because I see the reduce_left_padding function is called when preparing decode and not prefill?

[sducouedic]

Yes, you're right! I am wondering if it is the desired behaviour, and if something prevents to have the left padding already removed in the subsequent prefill. Conceptually it makes more sense to have the padding already removed if a prefill happens (ie. call reduce_left_padding in both decode and prefill). Also supposing the tkv is too close to max_model_len at the time of the prefill, the prompt would need to wait 1 decode, instead of being directly scheduled if the padding reduction happens directly.

[prashantgupta24]

Hmm, yeah, that does make sense to me.

[prashantgupta24]

Although it might be counter-intuitive since the left padding is introduced when you prepare a prefill for a new request? So ideally we could add logic while adding the padding instead of adding then reducing the padding if that makes sense

[sducouedic]

ah I see what you mean, that would be more code instead of just calling the reduce_left_padding function after adding the padding. Let's see what @yannicks1 has to say

[yannicks1]

I do agree and am working on reducing the left padding in every scheduler step (prefil and decode). Thanks for pointing this out!

[prashantgupta24]

LGTM. I guess the left padding question still remains.

[yannicks1]

This is really nice work @sducouedic. Super clean tests on scheduler step level. I suggest to merge this. I am already preparing the improved reducing of the left padding on another branch and would then update the tests accordingly.

[yannicks1]

I do agree and am working on reducing the left padding in every scheduler step (prefil and decode). Thanks for pointing this out!