Proposal: new chat_template_arg `enable_history_reasoning` for reusing prompt cache among querys within Agents .

[Abioy]

Description

Hi Qwen Team,

Not sure if this is the right place to post this proposal, but I don't know of anywhere else to post it and have a discussion.

I'm testing Qwen3.5 with some agents that involve many tool calls and multi-turn interactions. One thing catches my attention is that the prompt cache hit rate drops every time I send a new message.

I found this is because the chat template won't render the reasoning parts before the last user query message into the final prompt. Therefore, these messages cannot hit the prompt cache. The strategy could help in reducing context length and has little impact if there are few messages between two user queries.

But when it comes to the Deep Agent case, since tool calls are more often, the size of the reasoning content seems to be small (like tens of tokens?), making the overhead of re-processing these messages less acceptable.

So, I think it's better to offer a choice between reducing history context length and reducing prefill latency.

I've already submitted a PR to fix the chat template here: https://huggingface.co/Qwen/Qwen3.5-35B-A3B/discussions/60

I'd appreciate your feedback when you have a chance. Thanks!

Reproduction

enable thinking in some agent.
send a new mesage.

Logs

Environment Information

Known Issue

• The issue hasn't been already addressed in Documentation, Issues, and Discussions.

[JJJYmmm]

Hi! Which inference engine are you using? qwen3.5 are hybrid models with linear attention states, so the cache can't be trimmed as the usual one. Some engines like llama.cpp/mlx supports checkpointing for qwen3.5, and it can also handle the lacked reasoning content in the history.

[Abioy]

Thx @JJJYmmm

Hi! Which inference engine are you using?

Both mlx and llama.cpp.

qwen3.5 are hybrid models with linear attention states, so the cache can't be trimmed as the usual one. Some engines like llama.cpp/mlx supports checkpointing for qwen3.5, and it can also handle the lacked reasoning content in the history.

Yes, llama.cpp has checkpoints and mlx recently added similar caching support. There's still a common prefix between the last request and some checkpoints. The part I intend to speed up is the content between two user query messages.

For example, each tool call response after round 1 below could be prefilled very fast when sent to Qwen3.5, because (almost) all messages before it are cached in the last checkpoint:

// round 1
user: msg no.1 do sth blah blah ...
assistant: reasoning no.1 + tool call no.1
tool: ...
assistant: tool call no.2
tool: ...
... <after a long loop> ...
assistant: tool call no.N
tool: ...
assistant: resp

But if I send another message, since the last user message index is no longer 0, all reasoning content won't be rendered, and thus messages after the first user message have to be reprocessed.

// round 1
user: msg no.1 do sth blah blah ...                // reprocess (prefill) after here
assistant: reasoning no.1 + tool call no.1
tool: ...
assistant: tool call no.2
tool: ...
... <after a long loop> ...
assistant: tool call no.N
tool: ...
assistant: resp
// round 2
user: msg no.2 blah blah ...   // cache miss (except system msg, tools def, first user msg)

The issue is, if the long loop is long enough—for example, tens of tool calls—the prefill latency might be significant compared to the context-saving benefit.

[JJJYmmm]

okay, I got it, the checkpointing only save the prefix before the final token, and thus skip the last round of tool calls and results.
preserving the reasoning content can reuse the prefix cache directly, like clear_thinking in other agentic models.
https://huggingface.co/zai-org/GLM-5/blob/main/chat_template.jinja#L52-L53

Let's ping @jklj077 for the enhancement proposal.

[jklj077]

Hi, this feature will be included in upcoming releases (Qwen3.6).