Skip to content

Mang/eagle3 similarity - #38079

Draft
mangguo321 wants to merge 1 commit into
openvinotoolkit:masterfrom
mangguo321:mang/eagle3_similarity
Draft

Mang/eagle3 similarity#38079
mangguo321 wants to merge 1 commit into
openvinotoolkit:masterfrom
mangguo321:mang/eagle3_similarity

Conversation

@mangguo321

Copy link
Copy Markdown
Contributor

Details:

  • Root cause: EAGLE3 needs to reorder and compact the KV cache after it selects the valid tokens from a speculative token tree.
    Previously, PaKVReorder changed the Key and Value cache tensors in place, but exposed only a dummy output tensor. In other words, the graph did not explicitly say that the updated K/V caches were results of the operation.
    Because of this, the runtime could execute the reorder using temporary internal buffers. The reorder operation modified those buffers, but the updated data was not guaranteed to be written back to the external KV cache used by the next decoding step.
    As a result, EAGLE3 could select the correct branch but then continue decoding with stale cache data. This caused token divergence, especially in the FP16 configuration where a small cache difference can more easily change the next selected token.
  • Fix: We changed PaKVReorder to produce two real outputs:
    output 0: updated Key cache
    output 1: updated Value cache
    The CPU plugin marks each output as in-place with its corresponding input:
    Key cache input <-> Key cache output
    Value cache input <-> Value cache output
    Therefore, no additional full-cache copy is needed.
    We also changed the EAGLE3 helper graph and fusion pass so that Key and Value cache updates are separate model outputs. GenAI explicitly binds these outputs to the external KV cache tensors.
    Now the runtime knows that the reordered K/V caches are real results that must be preserved for the next decoding step. After the fix, FP16 EAGLE3 produces the same token IDs as target-only decoding in the tested deterministic cases, and the full WWB similarity improved from 0.88574980 to 0.97614926.

Tickets:

AI Assistance:

  • AI assistance used: no / yes
  • If yes, summarize how AI was used and what human validation was performed (build/tests/manual checks).

@github-actions github-actions Bot added category: inference OpenVINO Runtime library - Inference category: Core OpenVINO Core (aka ngraph) category: GPU OpenVINO GPU plugin category: CPU OpenVINO CPU plugin category: build OpenVINO cmake script / infra category: transformations OpenVINO Runtime library - Transformations category: CI OpenVINO public CI category: docs OpenVINO documentation category: dependency_changes Pull requests that update a dependency file category: CPP API OpenVINO CPP API bindings category: JS API OpenVino JS API Bindings category: NPU OpenVINO NPU plugin category: NPUW NPUW plugin labels Sep 11, 2026
@mangguo321
mangguo321 force-pushed the mang/eagle3_similarity branch from 6c90204 to 545b8fd Compare September 11, 2026 03:26
@github-actions github-actions Bot removed category: inference OpenVINO Runtime library - Inference category: GPU OpenVINO GPU plugin category: build OpenVINO cmake script / infra category: CI OpenVINO public CI category: docs OpenVINO documentation category: dependency_changes Pull requests that update a dependency file category: CPP API OpenVINO CPP API bindings category: JS API OpenVino JS API Bindings category: NPU OpenVINO NPU plugin category: NPUW NPUW plugin labels Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category: Core OpenVINO Core (aka ngraph) category: CPU OpenVINO CPU plugin category: transformations OpenVINO Runtime library - Transformations

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant