FR [v0.4.0?]: P/D Disaggregation related metrics

[JeffLuoo]

What would you like to be added:

Provide high-level metric:

• The # of requests that are routed to P/D given the threshold.

Why is this needed:

• Allow users to tune the threshold in the configuration of the plugin to decide that, given the current available hardware resources allocated to prefill and decode workers, how many request are they willing to enable the P/D?

Need more metrics?:

Please feel free to drop a comment if you think there are other useful metrics needed.

[JeffLuoo]

Metrics package in GIE is https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/epp/metrics/metrics.go.

Q: What's the most convenient way to extend the scheduler plugin to allow metrics defined in plugins accessible in the EPP? Ideally, I would like to only have a single /metrics endpoint to scrape the metric.

[nirrozenbaum]

Metrics package in GIE is https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/epp/metrics/metrics.go.

Q: What's the most convenient way to extend the scheduler plugin to allow metrics defined in plugins accessible in the EPP? Ideally, I would like to only have a single /metrics endpoint to scrape the metric.

the right approach would probably be to use the new data layer (currently experimental) and then add extended attributes.

cc @elevran for more details

[JeffLuoo]

@nirrozenbaum Thanks. Do you have any pointer to the new data layer (proposal / doc)? When will it become available to use?

[nirrozenbaum]

more information here:
https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main/docs/proposals/1023-data-layer-architecture

I’ll let Etai answer about timeline as he is driving the implementation.

[elevran]

The new data layer is used to collect additional data for scheduling and not publish additional metrics from the scheduler.

@JeffLuoo - is the intent to add a new metric to the EPP (in llm-d) that shows percent/count of request that were assigned disaggregated P/D?
If so, I think the issue is about adding a new metric to the /metrics endpoint for llm-d-inference-scheduler.
I think we might be missing a clean way to pass additional metrics to metrics.Register(customCollectors ...prometheus.Collector) since it is currently called from runner.go directly and not exposed to llm-d-inference-scheduler's main.go.
A possible workaround would be to import sigs.k8s.io/controller-runtime/pkg/metrics into the plugin and call metrics.Register for new metrics that are needed. Ultimately, we should provide llm-d-inference-scheduler pluging with access to IGW's runner,go metrics registration function.
cc: @nirrozenbaum

[JeffLuoo]

is the intent to add a new metric to the EPP (in llm-d) that shows percent/count of request that were assigned disaggregated P/D?

Yes.

A possible workaround would be to import sigs.k8s.io/controller-runtime/pkg/metrics into the plugin and call metrics.Register for new metrics that are needed

That's a good workaround as a short-term solution. However, for long-term solution, is such use case of extending the EPP to accept metric registration from extensions part of https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main/docs/proposals/1023-data-layer-architecture?

[JeffLuoo]

I have sent a PR in EPP: kubernetes-sigs/gateway-api-inference-extension#1787

And how metric will be added in the llm-d scheduler: #405

Note that, the draft PR is just for testing, not the real metric.

@nirrozenbaum @elevran PTAL and let me know what you think

[nirrozenbaum]

Thanks @JeffLuoo. overall lgtm.
I approved kubernetes-sigs/gateway-api-inference-extension#1787.
for #405 it looks half baked draft so haven’t added any comments yet.
we would probably need to better structure things, requests count could be reported in profile picker plugin where we decide if pd is triggered or not.