Sleep mode support #584
Sleep mode support #584
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR implements sleep mode support for HPU workers in vLLM, enabling memory reduction by moving models to CPU and clearing KV cache when workers are idle.
- Adds
sleep()andwake_up()methods toHPUWorkerclass for memory management - Enables sleep mode availability on the HPU platform
- Reorganizes import statements for better code organization
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| vllm_gaudi/v1/worker/hpu_worker.py | Implements sleep/wake_up methods with model CPU offloading and KV cache management, adds state tracking flags |
| vllm_gaudi/platform.py | Declares sleep mode support available on HPU platform |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
Kacper-Pietkun
force-pushed
the
dev/kpietkun/sleep_mode
branch
from
1a94591 to
d6922a2
Compare
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
Kacper-Pietkun
force-pushed
the
dev/kpietkun/sleep_mode
branch
from
d6922a2 to
2d629d0
Compare
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
Signed-off-by: Kacper Pietkun <[email protected]>
Kacper-Pietkun
force-pushed
the
dev/kpietkun/sleep_mode
branch
from
2d629d0 to
43c9d88
Compare
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
Signed-off-by: Kacper Pietkun <[email protected]>
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
Signed-off-by: Kacper Pietkun <[email protected]>
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
Kacper-Pietkun
requested review from
adobrzyn,
afierka-intel,
iboiko-habana,
kzawora-intel,
mgawarkiewicz-intel,
michalkuligowski,
vivekgoe and
xuechendi
as code owners
|
/run-gaudi-tests |
Signed-off-by: Kacper Pietkun <[email protected]>
|
/run-gaudi-tests |
✅ CI PassedAll checks passed successfully against the following vllm commit: |
|
/run-gaudi-tests |
1 similar comment
|
/run-gaudi-tests |
✅ CI PassedAll checks passed successfully against the following vllm commit: |
Signed-off-by: Kacper Pietkun <[email protected]>
Kacper-Pietkun
force-pushed
the
dev/kpietkun/sleep_mode
branch
from
c8969d7 to
2314cce
Compare
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
|
/run-gaudi-tests |
Kacper-Pietkun
force-pushed
the
dev/kpietkun/sleep_mode
branch
from
007dc3d to
2314cce
Compare
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
|
/run-gaudi-tests |
Signed-off-by: Kacper Pietkun <[email protected]>
Kacper-Pietkun
force-pushed
the
dev/kpietkun/sleep_mode
branch
from
c930e9c to
8f10fcd
Compare
|
/run-gaudi-tests |
✅ CI PassedAll checks passed successfully against the following vllm commit: |
tests/full_tests/ci_tests.sh
Outdated
| exit -1 | ||
| fi | ||
| echo "Test with sleep mode passed" | ||
|
|
There was a problem hiding this comment.
Hi, @Kacper-Pietkun , may you move it to ci_gsm8k_tests.sh => Sorry for the very confusing name, but only funcs inside ci_gsm8k_tests.sh will be auto detected by our CI
Signed-off-by: Kacper Pietkun <[email protected]>
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
Signed-off-by: Kacper Pietkun <[email protected]>
✅ CI PassedAll checks passed successfully against the following vllm commit: |
2 participants
Sleep mode level 1 - based on HabanaAI/vllm-fork#2055