[Bugfix][V0][V1] Fix crashes from cancelling requests

[tjohnson31415]

Some fixes from testing handling of request cancellation:

• in V0, guard against a KeyError in _req_ids2idx
• in v1, specialize the Scheduler's finish_requests() to handle the holdback_queue

FIX #36

[github-actions]

👋 Hi! Thank you for contributing to vLLM support on Spyre.
Just a reminder: Make sure that your code passes all the linting checks, otherwise your PR won't be able to be merged. To do so, first install the linting requirements, then run format.sh and commit the changes:

pip install -r requirements-lint.txt
bash format.sh

Now you are good to go 🚀

[joerunde]

Ah dang, this is unfortunate.

I think maybe we can fix this in a simpler way by removing self.holdback_queue as an instance attribute, and instead just make it a local variable during self.schedule(). After we schedule a new batch, we can take all the requests that we held back and put them back in self.waiting, and then we won't need to worry about breaking assumptions that the v1 scheduler has

[joerunde]

That would let us get rid of the override on get_num_unfinished_requests as well

[tjohnson31415]

Done!
I left it as an instance variable so that we don't remake the deque. It is also still used in _handle_rejects, though there probably can't be rejection during scheduling?

[joerunde]

Right, I think that it should be safe to remove it from usage in _handle_rejects as well since this should all be synchronous. The output processing that calls _handle_rejects can't be happening concurrently with scheduling a new forward pass

[joerunde]

I'll leave it up to you on merging now vs. pulling out of _handle_rejects as well. I think we'll be getting rid of this rejected request business soon anyway