Docs: Adding documentation for ScaleFromZero Feature - #647
Conversation
5269f20 to
9713d87
Compare
dumb0002
changed the title
There was a problem hiding this comment.
Pull request overview
This PR adds comprehensive user-facing documentation for the Scale-From-Zero feature in the Workload-Variant-Autoscaler (WVA). The feature enables automatic scaling of inactive model deployments (with 0 replicas) when there are pending inference requests, helping optimize resource utilization.
Changes:
- Added new documentation file
docs/user-guide/scale-from-zero.mdwith comprehensive coverage of the Scale-From-Zero feature, including architecture overview, configuration, usage examples, monitoring, and troubleshooting guidance
| name: epp-service | ||
| ``` | ||
|
|
||
| 4. **Send inference requests** - The Scale-From-Zero engine will automatically detect pending requests and scale up the deployment. |
There was a problem hiding this comment.
Can we clarify which deployment will be scaled up from multiple deployments available for a single base model?
There was a problem hiding this comment.
Right now we are scaling all the deployment associated inactive VA for the same target model. I have on my TODO list to improve this feature to scale only the VA that has the lowest cost.
There was a problem hiding this comment.
Maybe explicitly mentioned the same? or link a GitHub issue in the doc
There was a problem hiding this comment.
I will open a GitHub issue but adding a link to a Git issue or providing details, wouldn't be too much information to include into an user-guide doc? IMO that info is relevant for an admin or an advanced user.
| 1. **High concurrent processing load**: | ||
|
|
||
| **Solution**: Increase `SCALE_FROM_ZERO_ENGINE_MAX_CONCURRENCY`: | ||
| ```yaml | ||
| env: | ||
| - name: SCALE_FROM_ZERO_ENGINE_MAX_CONCURRENCY | ||
| value: "50" |
There was a problem hiding this comment.
What is the relation between high concurrency and slow scale-up from zero?
There was a problem hiding this comment.
If there are several deployments that needs to be scale up from zero, the high concurrency will speed up the process.
There was a problem hiding this comment.
ok, this is just the signal to scale up that is faster, clarify the same.
There was a problem hiding this comment.
Got it! I added more explanation in the troubleshooting.md clarify the use-case to change the values of this variable. This section is now moved to the file developer-guide/troubleshooting.md.
| value: "50" | ||
| ``` | ||
|
|
||
| 2. **Prometheus query latency**: |
There was a problem hiding this comment.
Hmm, where do we use prom?, I thought we were doing direct metric scraping
There was a problem hiding this comment.
Yes! you're right, we are doing a direct metric scraping indeed. Removing that section. Good catch. Thanks!
|
|
||
| | Variable | Description | Default | Valid Range | | ||
| |----------|-------------|---------|-------------| | ||
| | `SCALE_FROM_ZERO_ENGINE_MAX_CONCURRENCY` | Maximum number of concurrent variant processing operations | `30` | Integer > 0 | |
There was a problem hiding this comment.
This is an advance feature for the administrator. I would probably move this under an advance section, for administrator.
There was a problem hiding this comment.
Ok, I will remove from this doc. There aren't enough content in this doc to create a new advance section for administrator.
2e85f32 to
62666e8
Compare
|
@lionelvillard @asm582, I addressed all comments above. This PR is ready for another round of review. |
|
|
||
| **Solution**: Verify EndpointPicker service is running and metrics are being collected. | ||
|
|
||
| 4. **No pending requests in queue**: |
There was a problem hiding this comment.
Can you include the command to query metrics?
There was a problem hiding this comment.
It's already included: inference_extension_flow_control_queue_size{target_model_name="meta/llama-3.1-8b"}
There was a problem hiding this comment.
this is not a CLI command. A curl command is what I meant.
|
|
||
| **Possible Causes**: | ||
|
|
||
| 1. **High concurrent processing load**: |
There was a problem hiding this comment.
I would explain when this can happen: when there are many variants that are scaled down to zero.
There was a problem hiding this comment.
Thanks! Just added this suggested explanation.
| value: "50" | ||
| ``` | ||
|
|
||
| 2. **Inference gateway not receiving requests**: |
There was a problem hiding this comment.
looks like a generic problem not related to slow scale-up.
There was a problem hiding this comment.
Yes, but IMO it's important to remind the user to debug it. We both came across this issue during testing our e2e tests.
|
|
||
| ## Prerequisites | ||
|
|
||
| - WVA and llm-d installed and running - deployment options avaiable for [kind](https://github.com/dumb0002/workload-variant-autoscaler/blob/main/deploy/kind-emulator/README.md), [OpenShift](https://github.com/dumb0002/workload-variant-autoscaler/blob/main/deploy/openshift/README.md) and [Kubernetes](https://github.com/dumb0002/workload-variant-autoscaler/blob/main/deploy/kubernetes/README.md) |
There was a problem hiding this comment.
nit: avaiable -> available
| containers: | ||
| - name: manager | ||
| env: | ||
| - name: SCALE_FROM_ZERO_ENGINE_MAX_CONCURRENCY |
There was a problem hiding this comment.
most admins won't need to change this value. Written like this it gives the impression that this is a required step to go through the guide. So many add a note saying this is an advanced setting and there is no need to change it in most cases.
There was a problem hiding this comment.
Ok, I see! I prefer to remove this section instead to avoid any confusion. The use-case for setting this variable is already explained in the troubleshooting.md file.
9c1c511 to
1231da3
Compare
|
@lionelvillard, I finished addressing your latest comments. This PR is ready for another round of review. Thanks! |
| # Troubleshooting | ||
|
|
||
|
|
||
| ## ScaleFromZero Guides |
There was a problem hiding this comment.
I would remove this as this troubleshooting doc is meant to be guide-agnostic
f751c42 to
99b5107
Compare
|
@lionelvillard, addressed all the latest comment in this PR. Thanks! |
759bc08 to
9674f93
Compare
07c5e42 to
c0f535a
Compare
Signed-off-by: Braulio Dumba <Braulio.Dumba@ibm.com>
c0f535a to
4751956
Compare
lionelvillard
left a comment
lionelvillard
left a comment
There was a problem hiding this comment.
LGTM
@asm582 for approval as he added some comments.
This PR is extending the WVA user-guide with documentation for the or ScaleFromZero Feature