Documents a bit CB script and tests

[sducouedic]

This PR introduces a brief overview on how to debug and test the continuous batching functionality in vLLM. It pinpoints the main testing functions and script for inference with continuous batching.

[github-actions]

👋 Hi! Thank you for contributing to vLLM support on Spyre.
Just a reminder: Make sure that your code passes all the linting checks, otherwise your PR won't be able to be merged. To do so, first install the linting requirements, then run format.sh and commit the changes. This can be done with uv directly:

uv sync --frozen --group lint --active --inexact

Or this can be done with pip:

uv pip compile --group lint > requirements-lint.txt
pip install -r requirements-lint.txt
bash format.sh

Now you are good to go 🚀

[rafvasq]

Quick note that the doc needs to be included in .nav to be shown on the site, see #226.

Also wonder if it fits better under Developer Guide and not User Guide since it seems primarily about debugging and testing.

[prashantgupta24]

I wonder if all this can be added as docstring to the top of the respective test files themselves instead of having it sit separately (which is prone to being stale and an extra step for anyone trying to understand the code)

[sducouedic]

Having one doc file encompassing the different CB testing/debugging functions is a request from the compiler team actually

[prashantgupta24]

Makes sense. I wonder if we can add all the text from this PR into the test files and have some sort of table/visual representation encompassing the various configurations tested within the docs?

I worry that when we change something for the CB tests (which happens so frequently these days), we will forget to update the docs for the same which will lead to outdated stuff pretty fast.

[sducouedic]

The table/visual is a good idea, and actually also a request :)
But do you have in mind an automation code that update the table/visuals in the docs given the parameters found in the tests functions? also in your mind the table/visual would be in addition to the text in the docs, or replacing the text?

[prashantgupta24]

Ideally the visuals would be in addition to the docstring but I don't know if this can be automated, seems complex.

[rafvasq]

Just to jump in, I also agree that a lot of the info here probably should be just contained in the files too and this CB developer doc could point to where the script and tools are, with high-level descriptions, along with debugging notes similar to the ones here. Including that table you both mentioned.

I also noticed some "TODO" type of notes in this guide too. Those should probably be made issues for tracking rather than notes in the docs if they're important

[prashantgupta24]

Do we still need this PR?

[sducouedic]

@prashantgupta24
I just shortened it and added links to the docstring, please check: https://vllm-spyre--300.org.readthedocs.build/en/300/contributing/continuous_batching/overview.html

[yannicks1]

lgtm

[rafvasq]

Suggested change

	# Continuous Batching tests / inference scripts in vLLM
	Brief overview of what has been implemented so far in VLLM to test / debug continuous batching
	# Continuous Batching Testing and Debugging
	Overview of current tools for testing and debugging continuous batching in vLLM.

[rafvasq]

Suggested change

	## CB tests through unit tests
	## Unit Tests

[rafvasq]

These could be hyperlinked to the scripts in the repo

[rafvasq]

Suggested change

	* Runs inference on a set of prompts with continuous batching enabled (number of prompts is parametrizable)
	* Runs inference on a set of prompts with continuous batching enabled

[rafvasq]

Suggested change

	* All the requested sequences are defined in the beginning, there is no requests joining the waiting queue while the decoding of some other request has already started.
	* All requested sequences are defined at the start; no new requests join the queue once decoding has started for others.

[rafvasq]

Suggested change

	* If `--compare-with-CPU` is set, then the output text is compared to the one of hugging face, running on CPU. Note that here the logprobs are not compared, only tokens.
	* If `--compare-with-CPU` is set, the output text is compared to that of Hugging Face running on CPU. Note that only the tokens are compared, logprobs are not.
	* ```

[rafvasq]

Suggested change

	Output tests checks the correctness of the output of CB on a set of prompts. For now, the number of prompts and the prompts themself are hardcoded, as well as the max requested tokens per prompt (constant and set to 20). The output from vllm is compared to this of Hugging Face on CPU.
	Output tests check the correctness of the output of CB on a set of prompts. For now, the number of prompts and the prompts themself are hardcoded, as well as the max requested tokens per prompt (constant and set to 20). The output from vLLM is compared to that of Hugging Face on CPU.

[rafvasq]

Ah, I reviewed too late 😅. I can offer some of these updates myself later.

[sducouedic]

that would be great @rafvasq thanks