Skip to content

[CB] Update CB docs + Refactoring scheduling step-by-step inference tests #323

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Jul 18, 2025
Merged

[CB] Update CB docs + Refactoring scheduling step-by-step inference tests #323

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
90f9fb1
make 'get_params' functions unit tests
sducouedic Jul 17, 2025
fd88dc2
move functions to new dedicated files
sducouedic Jul 17, 2025
b18c4ad
rename 'test_scheduler_cb_steps_tkv' and deconnect it from CB
sducouedic Jul 17, 2025
8167860
rename file
sducouedic Jul 17, 2025
1057bd2
rename unit tests to be more informative
sducouedic Jul 17, 2025
3f1e91a
indentation
sducouedic Jul 17, 2025
b094874
test docs rendering
sducouedic Jul 17, 2025
b040ffb
add new tests docs built from docstring
sducouedic Jul 17, 2025
48940a6
Add configuration in docstring
sducouedic Jul 17, 2025
c4c8ca8
fix formatting
sducouedic Jul 17, 2025
6e7ea76
tests actually need to be prefixed with 'test'
sducouedic Jul 17, 2025
e00be31
put 'how to disable-asserts' tip in doc instead of tests docstring
sducouedic Jul 18, 2025
898b869
add all tests marked with 'cb' to 'other_tests'
sducouedic Jul 18, 2025
ab38ddd
rename end-to-end tests to output tests
sducouedic Jul 18, 2025
2b61721
rename docs scheduler tests part
sducouedic Jul 18, 2025
0b434b2
reformat differently formatted
sducouedic Jul 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions docs/.nav.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,10 @@ nav:
- Developer Guide:
- Contributing: contributing/README.md
- Continuous Batching:
- Testing: contributing/continuous_batching/tests.md
- Tests:
- End-to-End: contributing/continuous_batching/tests/e2e_testing.md
- Step-by-Step: contributing/continuous_batching/tests/scheduling_inference_steps.md
- Other Tests: contributing/continuous_batching/tests/other_tests.md

- Getting Started:
- Installation: getting_started/installation.md
Expand All @@ -34,4 +37,7 @@ nav:
- Developer Guide:
- Contributing: contributing/README.md
- Continuous Batching:
- Testing: contributing/continuous_batching/tests.md
- Tests:
- End-to-End: contributing/continuous_batching/tests/e2e_testing.md
- Step-by-Step: contributing/continuous_batching/tests/scheduling_inference_steps.md
- Other Tests: contributing/continuous_batching/tests/other_tests.md
3 changes: 0 additions & 3 deletions docs/contributing/continuous_batching/tests.md
View file
Open in desktop

This file was deleted.

10 changes: 10 additions & 0 deletions docs/contributing/continuous_batching/tests/e2e_testing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# End output correctness tests
Copy link
Collaborator

@yannicks1 yannicks1 Jul 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we find a better title for this section? It would be nice to coincide with the chapter on the left of the page. Currently we have:

End-to-End -> title: End output correctness tests -> file name: e2e_testing.md
Step-by-Step -> title: Scheduler inference steps tests -> file name: scheduling_inference_steps.md
Other Tests -> title: Other tests -> file name: other_tests.md

what about:

Output Tests -> title: Output Tests -> file name: output_tests.md
Scheduler Tests -> title: Scheduler Tests -> file name: scheduler_tests.md
Other Tests -> title: Other tests -> file name: other_tests.md

e.g. making things more uniform?

Copy link
Collaborator Author

@sducouedic sducouedic Jul 18, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure about scheduler_tests only because there is some other test test_cb_max_tokens in test_spyre_cb.py that is also testing the scheduler (but just the constraint), without really belonging to this test suite imo. I want to emphasis that this is somehow "step-by-step correct execution". And it won't be only about the scheduler correctness, on the compiler side it is also required that the compiled graph cab adapt to different scenarios of prompts_length/max_tokens/steps joining.

But I agree the current naming is a bit fuzzy, I changed it a lot

Copy link
Collaborator Author

@sducouedic sducouedic Jul 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But output_test is great naming, I change that

Copy link
Collaborator Author

@sducouedic sducouedic Jul 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the scheduler tests, I renamed everything to scheduler_steps tests 👍


!!! note
Unless otherwise specified, all the continuous batching tests are running with `max_model_len=256`

::: tests.e2e.test_spyre_basic
options:
members:
- test_output
- test_batch_handling
6 changes: 6 additions & 0 deletions docs/contributing/continuous_batching/tests/other_tests.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Other tests

!!! note
Unless otherwise specified, all the continuous batching tests are running with `max_model_len=256`

::: tests.e2e.test_spyre_cb
9 changes: 9 additions & 0 deletions docs/contributing/continuous_batching/tests/scheduling_inference_steps.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Scheduler inference steps tests

!!! note
Unless otherwise specified, all the continuous batching tests are running with `max_model_len=256`

!!! warning
End output correctness is not verified in those tests (TODO should we? maybe for some of them?)

::: tests.e2e.test_spyre_cb_inference_steps
24 changes: 18 additions & 6 deletions tests/e2e/test_spyre_basic.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
"""Verification of vLLM output by comparing with HF

If errors occur, these can be analyzed/debugged by setting
`DISABLE_ASSERTS = True` in spyre_util.py and by rerunning the
test using `pytest --capture=no tests/spyre/test_spyre_basic.py`
After debugging, `DISABLE_ASSERTS` should be reset to `False`.

Run `python -m pytest tests/e2e/test_spyre_basic.py`.
"""

Expand Down Expand Up @@ -45,11 +50,12 @@ def test_output(
The same prompts are also input to HF. The generated output
including text, token ids, and logprobs, is verified to be
identical for vLLM and HF.

If errors occur, these can be analyzed/debugged by setting
'DISABLE_ASSERTS = True' in spyre_util.py and by rerunning the
test using 'pytest --capture=no tests/spyre/test_spyre_basic.py'
After debugging, DISABLE_ASSERTS should be reset to 'False'.

Configuration for CB - parameters are combinatorial:
* max_num_seqs: 4
* tensor parallelism: 1, 2, 4, 8
* number of prompts: 4 (Chicken soup prompts)
* max tokens: 20 (same for all the prompts)
'''

skip_unsupported_tp_size(tp_size, backend)
Expand Down Expand Up @@ -156,7 +162,13 @@ def test_batch_handling(model: str, backend: str, cb: int,
monkeypatch: pytest.MonkeyPatch):
"""Test that the spyre worker correctly handles
continuous batches of requests that
finish after different numbers of forward passes"""
finish after different numbers of forward passes

Configuration for CB - parameters are combinatorial:
* max_num_seqs: 2
* number of prompts: 4 (Chicken soup prompts)
* max tokens: [5, 20, 10, 5]
"""

prompts = get_chicken_soup_prompts(4)

Expand Down
Loading
Loading