-
Notifications
You must be signed in to change notification settings - Fork 26
[SB][CB] mark input tensors refactoring #198
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
…hing case Signed-off-by: Yannick Schnider <[email protected]>
👋 Hi! Thank you for contributing to vLLM support on Spyre.
Or this can be done with
Now you are good to go 🚀 |
Signed-off-by: Yannick Schnider <[email protected]>
FYI: I did something similar in this other draft PR. My intention is try have common between static batching and continuous batching and in after that have a structure that is more similar to vLLM upstream. |
thanks for letting me know (I usually do not check draft PRs:) |
torch._dynamo.mark_static(model_input.slot_mapping, 1) # always 1 | ||
torch._dynamo.mark_static(model_input.input_positions, | ||
1) # always 1 | ||
self._mark_input_tensors(model_input) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we really need this function to be using self
? It only interacts with the model_input
param
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you are totally right, I copied it from the static batching code, where it also uses self which can be removed. FYI: @wallashss
If @wallashss can confirm above change I will close this PR and he can address all in his? |
I'm back to work in this refactoring, so sure I can address those. |
Thanks Wallas, closing this one now. |
WIP