Saved context state?

[sroussey]

Question

I have a function that is given a prompt and a list of tools, and then decides on which tool to call and how to form the inputs and then return the outputs.

So when I build up the context for this, I send a base system like prompt, and then the tool definitions, and then finally the user prompt.

So it needs to process all this for each tool call. I would like a checkpoint after my system prompt and tool definitions that I can "rewind" to for the next tool call. Is this possible? Either to rewind in memory or even to output somewhere and then reload (assuming that would be faster)?

[nico-martin]

What model do you use?

Unfortunately, this functionality is not available in the pipeline. However, if you call the model directly, you will receive the KV Cachepast_key_values of this request. The next time you make a follow-up request, i.e., the same tools, the same messages + a new message, you can pass the past_key_values. This means that the existing prefill tokens do not have to be calculated, which of course significantly improves performance.

[sroussey]

Using onnx-community/Qwen2.5-1.5B-Instruct at the moment. qwen3_5 is not supported by transformers.js

[xenova]

qwen3_5 is not supported by transformers.js

If you upgrade to @huggingface/transformers@next, you can use qwen3.5 :) (PR).

We also recently added support for cached generation (#1557) in Qwen-vl models

Demos:

• https://huggingface.co/spaces/webml-community/Qwen3.5-WebGPU (features pkv cached generation), probably what you want to look at
• https://huggingface.co/spaces/webml-community/Qwen3.5-0.8B-WebGPU, simpler demo

[sroussey]

@xenova -- even on next.6 I still get "Unsupported model type: qwen3_5"

edit: for model onnx-community/Qwen3.5-0.8B-ONNX

[xenova]

Just a reminder that you can't currently use it via the pipeline API (yet). You would need to use it as shown here: https://huggingface.co/onnx-community/Qwen3.5-0.8B-ONNX#transformersjs

[sroussey]

Ah, ok. Is there a tracking issue for that?

[xenova]

There is #1295, but ultimately we want to align with the transformers any-to-any pipeline API. image-text-to-text has been deprecated in the python transformers library.

[sroussey]

that makes sense. is there a qwen 3.5 model that is text only?

[xenova]

Nope, but with the next preview release, you'll be able to use qwen3.5 models via the text-generation pipeline as text-only models. PR: #1569