Add Node to update KV cache in Stateful LLM model

[Kotomi-Du]

Description

This PR is to add a small subgraph Gather + ScatterElementUpdate for KVCache to allow OpenVINO to do KV cache reorder during model inference. This pattern will be optimized out by OV GPU if there is no related information provided (done in OV 33114)

The graph below shows how the PR impacts an onnx model when triggering makeStateful() path.

Motivation and Context

The Microsoft Phi-Silica application leverages tree-based speculative decoding to accelerate LLM inference. This technique requires frequent manipulation of past KV cache states (e.g. trimming, reordering). This is because only a single branch of the speculative draft tree is accepted after verification.

On the other side, the current KV Cache API available is OV is very slow which cannot meet MSFT requirements. Details in CVS-174809. As OV team suggested, the only way to support reorder feature is to add specific nodes in the original graph. This PR is to serve this purpose.

Open

• If NPU don't want to have this path, a device specific flag has to be added.

If feature goes to new ABI?

Yes

Jira Ticket :

CVS-176367

[Copilot]

Pull request overview

This PR adds support for KV cache reordering in the OpenVINO stateful LLM model to enable tree-based speculative decoding. It introduces a new subgraph pattern (Gather + ScatterElementsUpdate) that allows OpenVINO to perform KV cache reordering during inference, which can be optimized out by the GPU if not needed.

Key changes:

• Adds new graph nodes (src_idx, dst_idx parameters and Gather/ScatterElementsUpdate operations) to enable KV cache manipulation
• Implements ReorderKVCache API across the backend stack with parsing logic for comma-separated index pairs
• Stores reorder indices in StatefulOVInferRequest for processing during inference

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
ov_stateful_patch_utils.h	Adds opset12 include for ScatterElementsUpdate operation
ov_stateful_patch_utils.cc	Implements the new KV cache reorder subgraph with src_idx/dst_idx parameters and Gather/ScatterElementsUpdate nodes
ov_interface.h	Declares ReorderKVCache method and adds member variables for storing reorder indices
ov_interface.cc	Implements ReorderKVCache with index validation and tensor population logic using hardcoded shape values
openvino_execution_provider.cc	Adds kvcache_reorder option parsing to convert semicolon-delimited string format into index vectors
ibackend.h	Adds virtual ReorderKVCache method to IBackend interface
basic_backend.h/cc	Implements ReorderKVCache to propagate calls to inference request pool
backend_manager.h/cc	Implements ReorderKVCache as pass-through to concrete backend

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[RyanMetcalfeInt8]

LGTM

[RyanMetcalfeInt8]

Thanks @Kotomi-Du & @ZackyLake, I've approved -- But please resolve one small typo for Lint warning above.

[Kotomi-Du]

Thanks @Kotomi-Du & @ZackyLake, I've approved -- But please resolve one small typo for Lint warning above.

Thanks, Ryan. We will address that, test the latest version again before merge!

[RyanMetcalfeInt8]

Thanks @Kotomi-Du & @ZackyLake, can you also address some of those other Lint warnings? I think you'll just need to include a couple more headers. e.g.

#include <vector>
#include <limits>

[Kotomi-Du]

Confirmed, Phi-silica tests on GPU are passed with latest change.

[Kotomi-Du]

Thanks @Kotomi-Du & @ZackyLake, can you also address some of those other Lint warnings? I think you'll just need to include a couple more headers. e.g.

#include <vector>
#include <limits>

Sorry, @RyanMetcalfeInt8 , I missed the comment you just updated and merged the PR. We can initialize a new PR for that or could address this warning in our future PRs.