Skip to content

Commit 2cdbea3

Browse files
jpr5jpr5
authored
Complete #274 blocks feature across all providers + blocks-only first-class + docs (#285)
## What Completes the (still-unreleased) #274 `blocks` fixture-ordering feature so it works **uniformly across every provider**, and makes blocks-only fixtures **first-class**. Builds on #283 (blocks v1) and #282 (async video), both already on `main` under `[Unreleased]`. No version bump — stays `1.34.0`. The dedicated `chore: release v1.35.0` PR owns the bump. ## Why #283 shipped `blocks` for a subset of providers and left the record-side capture + several provider builders out of scope, with blocks-only fixtures treated as by-design-unsupported. To ship the feature as *complete* for 1.35.0, it needs to: capture/replay block order on the remaining providers, accept a fixture that is *just* `{ blocks: [...] }`, and document the feature where authors actually look. ## Changes **Feature completion** - **First-class blocks-only fixtures** — a response may be `{ blocks: [...] }` with no `content`/`toolCalls`; the shared recognizer was relaxed *additively* (legacy recognition byte-identical). - **Record side** — block-order capture added to the Cohere, Bedrock (event-stream), and Gemini-Interactions collapsers; `normalizeToolArguments` zero-arg ("{}") fix. Gemini-Interactions is args-only by design (step-index can't reconcile arrival-order blocks). - **Replay builders** — Cohere, Bedrock (invoke), Bedrock-Converse, and Gemini-Interactions now honor `blocks` ordering (streaming + non-streaming where the wire allows). - **toolCall `arguments`** may be a JSON object or a string (objects auto-stringify), consistently across top-level toolCalls and block toolCalls. - **Validation hardening** — `validateBlocks` rejects empty-text blocks and warns on blocks/content divergence. **Completion gaps found + fixed during code review** (each with a local red→green on the real surface) - Ollama **non-streaming** dropped the entire payload for a blocks-only fixture (F0 made it newly reachable) — now backfills content/tool_calls from blocks. - Realtime + Gemini-Live WS surfaces silently dropped blocks-only payloads — now honor blocks. - Programmatic `addFixture` with object block args returned HTTP 500 (bypassed normalize) — `resolveFixtureBlocks` now tolerates object args. - A valid `{ content: "", blocks: [...] }` fixture spuriously hard-errored at load — suppressed when non-empty blocks drive output. - Gemini-Live empty-text block leaked past `truncateAfterChunks` and shifted recorded-timing indices — guard now skips emission cleanly. **Docs** - `docs/fixtures/index.html`: per-provider observability matrix, blocks-only authoring, reasoning cells. - Authoring on-ramps: `write-fixtures` skill/command, `docs/examples` (worked **tool-first** example), `docs/record-replay`, root + pytest READMEs. - A runnable `fixtures/examples/llm/blocks-tool-first.json` with a permanent loadability test. ## Wire limitations (documented, not gaps) OpenAI-chat, Ollama, and Cohere **non-streaming** expose separate content/tool_calls fields, so order is not observable on the wire — payload is delivered, order is a no-op. Captured in the observability matrix. ## Test plan - Full suite: **144 files / 4244 tests** pass - `tsc --noEmit`, `eslint`, `prettier --check`, `build`, `test:exports` (node10/node16-CJS/ESM/bundler) all green - Every behavioral fix verified red→green on the real surface (live endpoint / real WS / real loader), independently re-confirmed in review ## Known non-blocking follow-ups Pre-existing items (predate this branch) and minor polish are tracked but intentionally out of scope: a shared stringify helper to de-dup the three normalize sites, JSON-serializability check for object args at load time, and assorted pre-existing collapser/request-converter edge cases.
2 parents 36abb2d + 17166ca commit 2cdbea3

40 files changed

Lines changed: 4401 additions & 363 deletions

‎CHANGELOG.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,10 @@
1111
- Optional `blocks` array on the combined `content` + `toolCalls` fixture shape lets a fixture express ordered text/tool-call blocks (`{type:"text",text}` | `{type:"toolCall",name,arguments,id?}`); when present it takes precedence over `{content, toolCalls}` for stream order, enabling tool-first and interleaved ordering. Legacy `{content, toolCalls}` fixtures are unchanged (#274)
1212
- All five providers stream combined responses in fixture block order: Anthropic, OpenAI Responses, and Gemini are fully observable; Ollama is best-effort (clients may reassemble positionally); OpenAI chat-completions emits in order but is degenerate (`delta.content`/`delta.tool_calls` are separate channels the client merges) (#274)
1313
- Recorder captures block order and persists `blocks` only when the recorded upstream stream was genuinely tool-first or interleaved; text-first streams keep the legacy `{content, toolCalls}` shape so golden recordings round-trip byte-identically (#274)
14+
- Blocks-only fixtures are first-class: a non-empty `blocks` array is a complete response shape on its own, with no `content`/`toolCalls` required — builders derive the aggregate from the blocks and `validateFixtures()` accepts the shape (#274)
15+
- Block ordering is now honored on replay across the remaining providers — Cohere (streaming), Bedrock invoke, Bedrock Converse, and Gemini Interactions — so a tool-first or interleaved fixture streams its tool call ahead of its text wherever the wire protocol can express it (#274)
16+
- Record-side block capture extends to the Cohere and Bedrock collapsers; Gemini Interactions normalizes tool-call arguments only and does not reorder blocks on capture (its step-index protocol can't reconcile arrival-order blocks at record time), while replay still honors a hand-authored `blocks` array (#274)
17+
- `validateBlocks` rejects a malformed `blocks` array at load time — non-array, non-object entries, a `type` other than `text`/`toolCall`, a non-string or empty-string text block, or a `toolCall` block missing a name or carrying non-JSON arguments — and warns when a fixture carries both `blocks` and divergent `content`/`toolCalls`, so a bad array never reaches a builder mid-dispatch (#274)
1418

1519
## [1.34.0] - 2026-06-24
1620

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,7 @@ Run them all on one port with `npx @copilotkit/aimock --config aimock.json`, or
6464
- **[Docker + Helm](https://aimock.copilotkit.dev/docker)** — Container image and Helm chart for CI/CD
6565
- **[Vitest & Jest Plugins](https://aimock.copilotkit.dev/test-plugins)** — Zero-config `useAimock()` with auto lifecycle and env patching
6666
- **[Response Overrides](https://aimock.copilotkit.dev/fixtures)** — Control `id`, `model`, `usage`, `finishReason` in fixture responses
67+
- **[Ordered Blocks](https://aimock.copilotkit.dev/fixtures#ordered-blocks)** — A `blocks` array streams text and tool calls in any order (tool-first or interleaved); blocks-only fixtures are first-class, and the recorder captures order from genuinely tool-first/interleaved streams
6768
- **[Streaming Usage Chunks](https://aimock.copilotkit.dev/streaming-physics)** — `stream_options.include_usage` support emits a final chunk with token counts, matching OpenAI's streaming usage protocol
6869
- **[Rate Limiting Headers](https://aimock.copilotkit.dev/chaos-testing)** — `x-ratelimit-*` headers on every response and `Retry-After` on 429 errors for testing retry/backoff logic
6970
- **Zero dependencies** — Everything from Node.js builtins

‎docs/examples/index.html‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -154,6 +154,33 @@ <h3>Sequential Responses</h3>
154154
}</code></pre>
155155
</div>
156156

157+
<h3>Tool-First Blocks</h3>
158+
<p>
159+
Plain <code>content</code> + <code>toolCalls</code> can't express a tool call
160+
<em>before</em> text, or text and tools interleaved in a specific order &mdash; the
161+
builders always emit text first. Use <code>blocks</code> to pin the exact emission order.
162+
A fixture can be <strong>blocks-only</strong> (<code>response: { blocks: [...] }</code>
163+
with no <code>content</code>/<code>toolCalls</code>), as shown here.
164+
</p>
165+
<div class="code-block">
166+
<div class="code-block-header">
167+
fixtures/examples/llm/blocks-tool-first.json <span class="lang-tag">json</span>
168+
</div>
169+
<pre><code>{
170+
<span class="key">"fixtures"</span>: [
171+
{
172+
<span class="key">"match"</span>: { <span class="key">"userMessage"</span>: <span class="str">"what's the weather in NYC?"</span> },
173+
<span class="key">"response"</span>: {
174+
<span class="key">"blocks"</span>: [
175+
{ <span class="key">"type"</span>: <span class="str">"toolCall"</span>, <span class="key">"name"</span>: <span class="str">"get_weather"</span>, <span class="key">"arguments"</span>: <span class="str">"{\"city\": \"NYC\"}"</span> },
176+
{ <span class="key">"type"</span>: <span class="str">"text"</span>, <span class="key">"text"</span>: <span class="str">"Let me check the weather in NYC for you."</span> }
177+
]
178+
}
179+
}
180+
]
181+
}</code></pre>
182+
</div>
183+
157184
<!-- ─── Protocol Configs ───────────────────────────────────── -->
158185

159186
<h2>Protocol Configs</h2>

‎docs/fixtures/index.html‎

Lines changed: 132 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -384,41 +384,76 @@ <h2 id="ordered-blocks">Ordered blocks (tool-first &amp; interleaved streaming)<
384384
<p>
385385
When <code>blocks</code> is present it <strong>takes precedence</strong> over the
386386
<code>content</code> and <code>toolCalls</code> fields for stream ordering: the blocks are
387-
streamed in array order. (Keep <code>content</code> and <code>toolCalls</code> populated
388-
as well &mdash; they remain the canonical aggregate for replay and for consumers that do
389-
not read <code>blocks</code>.) When <code>blocks</code> is <strong>absent</strong>, legacy
387+
streamed in array order. When <code>blocks</code> is <strong>absent</strong>, legacy
390388
<code>{ content, toolCalls }</code> fixtures stream exactly as before &mdash; text-first,
391389
byte-identical to prior releases. The field is purely additive.
392390
</p>
391+
392+
<h3 id="blocks-only">Blocks-only fixtures (first-class)</h3>
393+
<p>
394+
A fixture can be written with <strong>only</strong> a <code>blocks</code> array &mdash; no
395+
<code>content</code> or <code>toolCalls</code> needed. A non-empty
396+
<code>blocks</code> array is a first-class response shape: the builders derive the
397+
aggregate text and tool calls from the blocks themselves, and
398+
<code>validateFixtures()</code> accepts it without requiring the legacy fields. This is
399+
the cleanest way to author a tool-first or interleaved response &mdash; you express the
400+
order once, in one place, with no duplicated aggregate to keep in sync.
401+
</p>
393402
<div class="code-block">
394403
<div class="code-block-header">tool-first.json <span class="lang-tag">json</span></div>
395404
<pre><code>{
396-
<span class="prop">"content"</span>: <span class="str">"Here is the weather."</span>,
397-
<span class="prop">"toolCalls"</span>: [
398-
{ <span class="prop">"name"</span>: <span class="str">"get_weather"</span>, <span class="prop">"arguments"</span>: { <span class="prop">"city"</span>: <span class="str">"SF"</span> } }
399-
],
400405
<span class="prop">"blocks"</span>: [
401406
{ <span class="prop">"type"</span>: <span class="str">"toolCall"</span>, <span class="prop">"name"</span>: <span class="str">"get_weather"</span>, <span class="prop">"arguments"</span>: { <span class="prop">"city"</span>: <span class="str">"SF"</span> }, <span class="prop">"id"</span>: <span class="str">"call_1"</span> },
402407
{ <span class="prop">"type"</span>: <span class="str">"text"</span>, <span class="prop">"text"</span>: <span class="str">"Here is the weather."</span> }
403408
]
404409
}</code></pre>
405410
</div>
406411
<p>
407-
The example above streams the <code>get_weather</code> tool call <em>before</em> the text.
408-
For an interleaved stream, list blocks in the desired order, e.g.
409-
<code>[toolCall, text, toolCall]</code>.
412+
The example above streams the <code>get_weather</code> tool call <em>before</em> the text,
413+
with no separate <code>content</code> / <code>toolCalls</code> fields. For an interleaved
414+
stream, list blocks in the desired order, e.g. <code>[toolCall, text, toolCall]</code>.
410415
</p>
416+
<p>
417+
You may still supply <code>content</code> and <code>toolCalls</code> alongside
418+
<code>blocks</code> if you want an explicit aggregate &mdash; for example to assert a
419+
specific merged shape independently of the order. Both forms are supported;
420+
<code>blocks</code> always wins for stream ordering.
421+
</p>
422+
<div class="info-box">
423+
<p>
424+
<strong>Validation:</strong> <code>validateFixtures()</code> checks a
425+
<code>blocks</code>
426+
array at load time so a malformed array is rejected before it reaches a builder &mdash;
427+
<code>blocks</code> must be an array; each entry must be an object with
428+
<code>type</code> <code>"text"</code> or <code>"toolCall"</code>; a
429+
<code>text</code> block needs a non-empty string <code>text</code>; a
430+
<code>toolCall</code> block needs a non-empty <code>name</code>,
431+
<code>arguments</code> that are a valid-JSON string or an object, and an optional string
432+
<code>id</code>. If a fixture carries both <code>blocks</code> and legacy
433+
<code>content</code>/<code>toolCalls</code> that disagree, loading warns (the redundant
434+
legacy fields are ignored in favor of <code>blocks</code>).
435+
</p>
436+
</div>
411437

412438
<h3>Per-provider observability</h3>
413439
<p>
414440
How faithfully &ldquo;tool-first&rdquo; / interleaved order is observable depends on each
415-
provider's wire protocol. The mock always emits chunks in block order; what a client can
416-
<em>reconstruct</em> from those chunks varies:
441+
provider's wire protocol &mdash; and, for some providers, on whether the request is
442+
streaming. The mock always emits in block order; what a client can
443+
<em>reconstruct</em> from the result varies. A shape is <strong>Full</strong> when the
444+
wire carries the blocks in a single positionally-ordered structure (indexed content
445+
blocks, ordered <code>output</code> items, ordered steps); it is
446+
<strong>Non-observable</strong> when text and tool calls land in
447+
<em>separate</em> top-level fields that the client merges without a shared order. It is
448+
<strong>Partial</strong> when block order <em>is</em> carried on the wire (chunk arrival
449+
order) but the structure is not positionally indexed, so some clients reassemble
450+
positionally rather than honoring arrival order &mdash; observable best-effort, not
451+
guaranteed. The classifications below were verified against each provider's builder.
417452
</p>
418453
<table class="endpoint-table">
419454
<thead>
420455
<tr>
421-
<th>Provider</th>
456+
<th>Provider / shape</th>
422457
<th>Block-order support</th>
423458
<th>Notes</th>
424459
</tr>
@@ -429,7 +464,8 @@ <h3>Per-provider observability</h3>
429464
<td>Full</td>
430465
<td>
431466
Typed <code>text</code> / <code>tool_use</code> content blocks at incrementing
432-
indices &mdash; tool-first and interleaved are natively observable.
467+
indices &mdash; tool-first and interleaved are natively observable, streaming and
468+
non-streaming alike.
433469
</td>
434470
</tr>
435471
<tr>
@@ -450,7 +486,45 @@ <h3>Per-provider observability</h3>
450486
</td>
451487
</tr>
452488
<tr>
453-
<td>Ollama</td>
489+
<td>Gemini Interactions (replay)</td>
490+
<td>Full</td>
491+
<td>
492+
One step per block in array order &mdash; a <code>function_call</code> step takes a
493+
lower <code>index</code> than a later <code>model_output</code> step, streaming
494+
(<code>step.*</code> events) and non-streaming (<code>steps[]</code>) alike. Record
495+
side is args-normalization only &mdash; see the note below.
496+
</td>
497+
</tr>
498+
<tr>
499+
<td>Bedrock invoke</td>
500+
<td>Full</td>
501+
<td>
502+
Mirrors the Anthropic Messages content array: ordered
503+
<code>text</code> / <code>tool_use</code> entries non-streaming, indexed
504+
<code>content_block_*</code> events streaming &mdash; tool-first is wire-expressible
505+
on both.
506+
</td>
507+
</tr>
508+
<tr>
509+
<td>Bedrock Converse</td>
510+
<td>Full</td>
511+
<td>
512+
Positional <code>content[]</code> blocks non-streaming, indexed
513+
<code>contentBlock*</code> events (carrying <code>contentBlockIndex</code>)
514+
streaming &mdash; a <code>toolUse</code> can precede the text on both.
515+
</td>
516+
</tr>
517+
<tr>
518+
<td>Cohere (streaming)</td>
519+
<td>Full</td>
520+
<td>
521+
SSE emits <code>content-*</code> and <code>tool-call-*</code> events in block array
522+
order, each carrying an <code>index</code> &mdash; tool-first / interleaved is
523+
observable on the stream.
524+
</td>
525+
</tr>
526+
<tr>
527+
<td>Ollama (streaming)</td>
454528
<td>Partial</td>
455529
<td>
456530
A <code>tool_calls</code> chunk can be emitted before content on the wire, but some
@@ -459,12 +533,33 @@ <h3>Per-provider observability</h3>
459533
</tr>
460534
<tr>
461535
<td>OpenAI chat-completions</td>
462-
<td>Degenerate</td>
536+
<td>Non-observable</td>
537+
<td>
538+
<code>delta.content</code> and <code>delta.tool_calls</code> (streaming), or
539+
<code>message.content</code> and <code>message.tool_calls</code> (non-streaming),
540+
are separate channels/fields the client merges. The mock emits in block order and
541+
the streamed wire order is assertable, but the merge is <em>not</em> positionally
542+
interleaved, so tool-first is not semantically observable to clients on this
543+
channel.
544+
</td>
545+
</tr>
546+
<tr>
547+
<td>Cohere (non-streaming)</td>
548+
<td>Non-observable</td>
463549
<td>
464-
<code>delta.content</code> and <code>delta.tool_calls</code> are separate channels
465-
the client merges. The mock emits chunks in block order (and the wire order is
466-
assertable), but the merge is <em>not</em> positionally interleaved, so tool-first
467-
is not semantically observable to clients on this channel.
550+
The non-streaming body keeps text in <code>message.content[]</code> and tool calls
551+
in the separate <code>message.tool_calls[]</code> field &mdash; the relative order
552+
of a text vs. a toolCall block is not on the wire. Use the streaming shape when
553+
order matters.
554+
</td>
555+
</tr>
556+
<tr>
557+
<td>Ollama (non-streaming)</td>
558+
<td>Non-observable</td>
559+
<td>
560+
The aggregated reply carries <code>message.content</code> and
561+
<code>message.tool_calls</code> as separate fields &mdash; no positional ordering
562+
between a text and a toolCall block. Use the streaming shape when order matters.
468563
</td>
469564
</tr>
470565
</tbody>
@@ -476,7 +571,16 @@ <h3>Per-provider observability</h3>
476571
<em>genuinely</em> tool-first or interleaved (a tool-call delta arrives before the first
477572
content delta, or content arrives after a tool-call delta). Ordinary text-then-tools
478573
streams are saved in the legacy <code>{ content, toolCalls }</code> shape with no
479-
<code>blocks</code> key, so existing golden recordings round-trip byte-identically.
574+
<code>blocks</code> key, so existing golden recordings round-trip byte-identically. The
575+
Cohere and Bedrock collapsers capture block order this way alongside the original
576+
providers.
577+
</p>
578+
<p>
579+
<strong>Gemini Interactions</strong> is the exception: its record-side collapser
580+
normalizes tool-call arguments only and does not reorder blocks on capture &mdash; its
581+
step-index protocol can't reconcile arrival-order blocks at record time. Ordering is
582+
still honored on <em>replay</em> from a hand-authored <code>blocks</code> fixture; it is
583+
simply not reconstructed automatically from a recording.
480584
</p>
481585
</div>
482586

@@ -789,12 +893,12 @@ <h2>Provider Support Matrix</h2>
789893
<td>Yes</td>
790894
<td>Yes</td>
791895
<td>Yes</td>
792-
<td>&mdash;</td>
896+
<td>Record only<sup>&dagger;</sup></td>
897+
<td>Yes</td>
793898
<td>Yes</td>
794899
<td>Yes</td>
795900
<td>Yes</td>
796901
<td>Yes</td>
797-
<td>&mdash;</td>
798902
</tr>
799903
<tr>
800904
<td>Web Searches</td>
@@ -828,6 +932,11 @@ <h2>Provider Support Matrix</h2>
828932
<sup>*</sup> Azure inherits OpenAI&rsquo;s override support because Azure OpenAI routes
829933
through the OpenAI Chat Completions response format internally.
830934
</p>
935+
<p class="footnote">
936+
<sup>&dagger;</sup> Gemini Interactions captures reasoning on record (its collapser
937+
assembles <code>thought_summary</code> deltas into <code>reasoning</code>), but its replay
938+
builders do not re-emit reasoning, so a replayed turn carries none.
939+
</p>
831940
</main>
832941
<aside class="page-toc" id="page-toc"></aside>
833942
</div>

‎docs/record-replay/index.html‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -429,6 +429,25 @@ <h2>Stream Collapsing</h2>
429429
simple <code>{ content }</code> or <code>{ toolCalls }</code> fixture response.
430430
</p>
431431

432+
<h2 id="recording-block-order">Recording Block Order</h2>
433+
<p>
434+
When a recorded stream is <em>genuinely</em> tool-first or interleaved &mdash; a tool-call
435+
delta arrives before the first content delta, or content arrives after a tool-call delta
436+
&mdash; the collapser preserves that arrival order as a
437+
<a href="/fixtures#ordered-blocks"><code>blocks</code></a> array on the fixture. This
438+
works across OpenAI, Anthropic, Gemini, Ollama, Cohere, and Bedrock. Ordinary
439+
text-then-tools streams are saved in the legacy <code>{ content, toolCalls }</code> shape
440+
with no <code>blocks</code> key, so existing recordings round-trip byte-identically.
441+
</p>
442+
<p>
443+
<strong>Gemini Interactions is the exception:</strong> its record-side collapser
444+
normalizes tool-call arguments only and does not reorder blocks on capture. Ordering is
445+
still honored on replay from a hand-authored <code>blocks</code> fixture; it is simply not
446+
reconstructed automatically from a recording. See the
447+
<a href="/fixtures#ordered-blocks">per-provider observability matrix</a> for how
448+
faithfully block order is reconstructable on each provider's wire.
449+
</p>
450+
432451
<h2>Header Forwarding</h2>
433452
<p>
434453
When proxying to upstream providers, aimock forwards the original request's headers except

‎fixtures/examples/llm/blocks-tool-first.json‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
{
2+
"fixtures": [
3+
{
4+
"match": { "userMessage": "what's the weather in NYC?" },
5+
"response": {
6+
"blocks": [
7+
{ "type": "toolCall", "name": "get_weather", "arguments": "{\"city\": \"NYC\"}" },
8+
{ "type": "text", "text": "Let me check the weather in NYC for you." }
9+
]
10+
}
11+
}
12+
]
13+
}

0 commit comments

Comments
 (0)