feat: toolResultContains match gate on the last tool-result payload (#299)

## Why

The showcase mastra interrupt demos use a native suspend tool where pick
and cancel resume with the SAME tool_call_id; the requests differ only
inside the tool-result message content (`{"chosen_time": ...}` vs
`{"cancelled": true}`). Every existing JSON-expressible gate is
identical between the two requests, so first-match-wins replays the pick
confirmation after the user cancelled. `predicate` cannot help: the JSON
fixture loader has no way to express it.

## What

New `match.toolResultContains: string` gate: passes when the request's
LAST message is a `role: "tool"` result whose text content contains the
substring. Mirrors the existing `toolCallId` last-message rule so the
two stay consistent.

- `src/router.ts`: gate placed with the other shape predicates, right
after `toolCallId`
- `src/types.ts`: `FixtureMatch` + `FixtureFileEntry.match`
- `src/fixture-loader.ts`: `entryToFixture` carry-through; validation
(string, non-empty — empty substring would silently act as a catch-all);
userMessage dedup key; catch-all discriminator set
- `packages/aimock-pytest`: `toolResultContains` added to
`_MATCH_LEVEL_OPT_KEYS` (0.5.0)
- docs: fixtures + multi-turn pages, README, write-fixtures skill
- tests: approve/cancel discrimination sharing one toolCallId; substring
present/absent; no-tool-message request never matches; stale tool result
followed by a new user turn never matches; array-of-parts content; null
content; loader validation + dedup + carry-through

## Version

1.37.0 (changelog entry added with this PR's number in a follow-up
commit on this branch).

---

### Maintainer note — additional fixes folded in during review

Beyond the `toolResultContains` gate, this branch now also carries (all
under the single 1.37.0 release):

- **fix:** completed the duplicate-userMessage dedup key so
legitimately-distinct fixtures no longer emit spurious shadow warnings
(kind-aware matcher serialization; `predicate` keyed by identity).
- **fix:** guarded `recordedTimings.interChunkDelaysMs` against
non-array values — a malformed entry no longer throws and silently drops
every fixture in its file.
- **fix:** bumped the `aimock-pytest` default server pin to 1.37.0 so
the auto-download path actually supports the new kwarg the client
forwards.
- **ci:** the `release` script now runs `format:check` + `test:exports`
before publish.
- **docs:** corrected the match-field / dedup / `toolCallId` semantics
docs, and fixed the write-fixtures E2E lifecycle to use
`resetMatchCounts()` instead of a full `reset()` between tests.

(These consolidate PR #300, now closed, into this one release. CHANGELOG
updated accordingly.)

Author: jpr5

CHANGELOG.md

@@ -1,5 +1,17 @@
 # @copilotkit/aimock
 
+## [1.37.0] - 2026-07-15
+
+### Added
+
+- New `match.toolResultContains` string gate: passes when the request's LAST message is a `role: "tool"` result whose text content contains the substring (same last-message rule as `toolCallId`, and composable with it). Discriminates fixtures whose requests differ only inside the tool-result payload — e.g. a human-in-the-loop suspend tool where approve and cancel resume with the same `tool_call_id` but different result JSON. JSON-expressible, so fixture files can finally split those legs without a programmatic `predicate`. Validated at load time (non-empty string — an empty substring would silently act as a catch-all), included in the duplicate-userMessage dedup key and the catch-all discriminator set, and routed through the `aimock-pytest` match-level kwarg keys (aimock-pytest 0.5.0) (#299)
+
+### Fixed
+
+- The duplicate-userMessage shadow check no longer warns for fixtures that share a `userMessage` but are legitimately disambiguated by a matcher the dedup key previously omitted (`systemMessage`, `model`, `toolName`, `toolCallId`, `inputText`, `responseFormat`, `endpoint`). The dedup key now covers every discriminator the router actually matches on and serialises RegExp / array matchers kind-aware so they don't collide; a `predicate`-bearing fixture is keyed by identity so it is never a false duplicate. This only suppresses spurious warnings — the check never dropped or deduped fixtures (#299)
+- Bumped the `aimock-pytest` default server pin (`_version.py`) to 1.37.0 so the auto-download path (used when `AIMOCK_CLI_PATH` is unset) runs a server that supports the new `toolResultContains` match kwarg the client forwards (#299)
+- Guarded `recordedTimings.interChunkDelaysMs` against non-array values: a fixture whose `interChunkDelaysMs` is malformed (null, missing, or not an array) no longer throws during load and silently drops every fixture in that file — the value is coerced to an empty array (#299)
+
 ## [1.36.1] - 2026-07-14
 
 ### Fixed

README.md

@@ -50,7 +50,7 @@ Run them all on one port with `npx @copilotkit/aimock --config aimock.json`, or
 
 - **[Record & Replay](https://aimock.copilotkit.dev/record-replay)** — Proxy real APIs, save as fixtures, replay deterministically forever
 - **Timing-aware recording and replay** — Recorded fixtures capture per-frame arrival timestamps; replay uses recorded timings for approximate timing reproduction based on recorded TTFT and inter-frame cadence (replay chunk count may differ from recording — TTFT and average pace are preserved, not per-token fidelity) with configurable `--replay-speed` multiplier
-- **[Multi-turn Conversations](https://aimock.copilotkit.dev/multi-turn)** — Record and replay multi-turn traces with tool rounds; match distinct turns via `turnIndex`, `hasToolResult`, `toolCallId`, `sequenceIndex`, `systemMessage` (gate on host-supplied agent context), or custom predicates
+- **[Multi-turn Conversations](https://aimock.copilotkit.dev/multi-turn)** — Record and replay multi-turn traces with tool rounds; match distinct turns via `turnIndex`, `hasToolResult`, `toolCallId`, `toolResultContains` (gate on the tool-result payload), `sequenceIndex`, `systemMessage` (gate on host-supplied agent context), or custom predicates
 - **[12 providers across 14 API surfaces](https://aimock.copilotkit.dev/docs)** — OpenAI Chat, OpenAI Responses, OpenAI Realtime (GA + Beta shim), Claude, Gemini (REST + embedContent), Gemini Live, Gemini Interactions, Azure, Bedrock, Vertex AI, Ollama (chat + embeddings), Cohere (chat + embed), ElevenLabs TTS — full streaming support
 - **Multimedia APIs** — [image generation](https://aimock.copilotkit.dev/images) (DALL-E, Imagen), [image editing](https://aimock.copilotkit.dev/images) (/v1/images/edits), [text-to-speech](https://aimock.copilotkit.dev/speech) (OpenAI + ElevenLabs), [audio transcription](https://aimock.copilotkit.dev/transcription), [audio translation](https://aimock.copilotkit.dev/transcription) (/v1/audio/translations), [video generation](https://aimock.copilotkit.dev/video), [OpenRouter video generation](https://aimock.copilotkit.dev/openrouter-video) (/api/v1/videos with async job lifecycle), [Google Veo video generation](https://aimock.copilotkit.dev/veo-video) (:predictLongRunning + /v1beta/operations async lifecycle), [Grok Imagine video generation](https://aimock.copilotkit.dev/grok-video) (/v1/videos/generations with async job lifecycle), [fal.ai](https://aimock.copilotkit.dev/fal-ai) (image / video / audio with queue lifecycle)
 - **[MCP](https://aimock.copilotkit.dev/mcp-mock) / [A2A](https://aimock.copilotkit.dev/a2a-mock) / [AG-UI](https://aimock.copilotkit.dev/agui-mock) / [Vector](https://aimock.copilotkit.dev/vector-mock)** — Mock every protocol your AI agents use

docs/fixtures/index.html

@@ -97,6 +97,16 @@ <h2>Match Fields</h2>
                 <code>requestTransform</code> is set) or regex (pattern match)
               </td>
             </tr>
+            <tr>
+              <td>systemMessage</td>
+              <td>string | RegExp | string[]</td>
+              <td>
+                Match on the joined text of the request&rsquo;s system messages &mdash; string
+                (case-sensitive substring, or exact when <code>requestTransform</code> is set),
+                regex (pattern match), or an array of strings (all substrings must be present). Gate
+                a fixture on host-supplied context
+              </td>
+            </tr>
             <tr>
               <td>inputText</td>
               <td>string | RegExp</td>
@@ -110,6 +120,16 @@ <h2>Match Fields</h2>
                 The <code>onToolResult(id, response)</code> helper is sugar over this field
               </td>
             </tr>
+            <tr>
+              <td>toolResultContains</td>
+              <td>string</td>
+              <td>
+                Substring match on the text content of the last message when it is a
+                <code>role: "tool"</code> result. Discriminates fixtures whose requests differ only
+                inside the tool-result payload (e.g. approve vs cancel resuming with the same
+                <code>tool_call_id</code>). See <a href="/multi-turn">Multi-Turn Conversations</a>
+              </td>
+            </tr>
             <tr>
               <td>toolName</td>
               <td>string</td>
@@ -200,12 +220,16 @@ <h3>1. <code>userMessage</code> matches only the LAST user message</h3>
           tool-round idiom.
         </p>
 
-        <h3>2. <code>toolCallId</code> matches the LAST tool message</h3>
+        <h3>2. <code>toolCallId</code> matches the overall last message</h3>
         <p>
-          <code>toolCallId</code> is compared against the <code>tool_call_id</code> of the
-          <em>last <code>role: "tool"</code> message</em> in the request &mdash; regardless of
-          whether that&rsquo;s the overall last message. If no tool message is present in the
-          history, <code>toolCallId</code> never matches. See
+          <code>toolCallId</code> matches only when the request&rsquo;s
+          <em>overall last message</em> is a <code>role: "tool"</code> result whose
+          <code>tool_call_id</code> equals the given value. This mirrors the API contract: a tool
+          result is the last message only while the model is answering that tool round. If the last
+          message is anything else &mdash; a newer <code>user</code> turn, or no tool result at all
+          &mdash; <code>toolCallId</code> never matches, so a stale tool round can&rsquo;t shadow
+          <code>userMessage</code> matchers. This is the same overall-last-message rule as
+          <code>toolResultContains</code>. See
           <a href="/multi-turn">Multi-Turn Conversations</a> for the tool-round idiom.
         </p>
 
@@ -234,10 +258,10 @@ <h3>4. Substring by default, exact when a <code>requestTransform</code> is set</
         <p>
           If the router is configured with a <code>requestTransform</code> (typically used to strip
           dynamic data like timestamps or UUIDs from the request before matching), string
-          <code>userMessage</code> and <code>inputText</code> flip to strict equality
-          (<code>===</code>). The rationale: transforms normalize requests to a canonical form, and
-          once normalized, the sensible comparison is exact &mdash; substring matching on a
-          normalized string is more likely to hide bugs than catch flexible input.
+          <code>userMessage</code>, <code>systemMessage</code>, and <code>inputText</code> flip to
+          strict equality (<code>===</code>). The rationale: transforms normalize requests to a
+          canonical form, and once normalized, the sensible comparison is exact &mdash; substring
+          matching on a normalized string is more likely to hide bugs than catch flexible input.
         </p>
 
         <h3>5. Validation warnings surface shadowing at load time</h3>
@@ -252,15 +276,19 @@ <h3>5. Validation warnings surface shadowing at load time</h3>
             <code>duplicate userMessage 'hello' — shadows fixture 0</code>, where
             <code>'hello'</code> is the duplicated message and <code>0</code> is the zero-based
             index of the earlier fixture being shadowed. This is advisory, not a hard error: the
-            check now factors in <code>turnIndex</code>, <code>hasToolResult</code>,
-            <code>context</code>, and <code>sequenceIndex</code> when deciding whether two fixtures
-            truly collide, but it does <em>not</em> consider <code>toolCallId</code>,
-            <code>model</code>, or <code>predicate</code>, so the warning may still fire when those
-            discriminators are present. Treat it as advisory: if a runtime differentiator is in
-            place, the fixtures won't actually shadow each other at match time. Only fixtures with
-            no differentiator at all will truly shadow on match &mdash; that's the case where the
-            second is never reached because the first wins. Safe to ignore in the former case;
-            investigate in the latter.
+            check factors in <em>every</em> match discriminator the router gates on when deciding
+            whether two fixtures truly collide &mdash; <code>userMessage</code>,
+            <code>systemMessage</code>, <code>inputText</code>, <code>toolCallId</code>,
+            <code>toolResultContains</code>, <code>toolName</code>, <code>model</code>,
+            <code>responseFormat</code>, <code>endpoint</code>, <code>context</code>,
+            <code>sequenceIndex</code>, <code>turnIndex</code>, and
+            <code>hasToolResult</code> &mdash; so two fixtures that differ on any one of them do not
+            trigger the warning. A <code>predicate</code> is a function that cannot be compared for
+            equivalence, so any fixture carrying one is keyed by its own identity: it is treated as
+            unconditionally distinct and never reported as a duplicate (and two fixtures with
+            distinct predicates are likewise treated as distinct). Only fixtures with no
+            differentiator at all will truly shadow on match &mdash; that's the case where the
+            second is never reached because the first wins. That is the case worth investigating.
           </li>
           <li>
             <strong>Catch-all not last</strong> &mdash; a fixture with an empty <code>match</code>

docs/multi-turn/index.html

@@ -72,6 +72,17 @@ <h2>How matching works across turns</h2>
             distinguish the turn that <em>requests</em> a tool from the turn that
             <em>follows up</em> on a tool result.
           </li>
+          <li>
+            <code>toolResultContains</code> (v1.37.0) is a substring match against the
+            <strong>text content of the last tool message</strong> (same last-message rule as
+            <code>toolCallId</code>). Use it when two resume paths share the same
+            <code>tool_call_id</code> and differ only inside the tool-result payload &mdash; e.g. a
+            human-in-the-loop tool where approve resumes with
+            <code>{"chosen_time": &hellip;}</code> and cancel resumes with
+            <code>{"cancelled": true}</code>. Gate the cancel fixture on
+            <code>"toolResultContains": "\"cancelled\""</code> (match the quoted JSON key so it
+            can&rsquo;t match the word inside prose) and order it before the approve leg.
+          </li>
         </ul>
         <p>
           A request carrying a 20-message history still only matches on its last user message (and,
@@ -100,8 +111,10 @@ <h2>How matching works across turns</h2>
             <strong>Substring by default, exact when transformed.</strong>
             <code>userMessage</code> is a substring match by default (<code>"hello"</code> matches
             <code>"say hello world"</code>). When you register a <code>requestTransform</code>,
-            matching flips to <strong>exact string equality</strong> &mdash; but only for
-            <code>userMessage</code> and <code>inputText</code>; other fields like
+            matching flips to <strong>exact string equality</strong> &mdash; but only for the
+            <em>string</em> forms of <code>userMessage</code>, <code>systemMessage</code>, and
+            <code>inputText</code>. Regex and array-of-substrings forms are unaffected, and
+            <code>toolResultContains</code> stays a substring match. Fields like
             <code>toolName</code> and <code>toolCallId</code> are always exact. This trips people up
             &mdash; see <a href="#gotchas">Gotchas</a> below.
           </p>
@@ -292,10 +305,10 @@ <h3>Turn 2 &mdash; client runs the tool, sends the result</h3>
         </div>
 
         <h2>
-          Choosing between sequenceIndex, toolCallId, turnIndex, hasToolResult, context, and
-          predicate
+          Choosing between sequenceIndex, toolCallId, toolResultContains, turnIndex, hasToolResult,
+          context, and predicate
         </h2>
-        <p>Five mechanisms handle different shapes of &ldquo;the same prompt twice&rdquo;:</p>
+        <p>Seven mechanisms handle different shapes of &ldquo;the same prompt twice&rdquo;:</p>
 
         <table class="endpoint-table">
           <thead>
@@ -318,8 +331,20 @@ <h2>
               <td>Different behavior before vs. after tool execution (tool-call round trip)</td>
               <td><code>toolCallId</code></td>
               <td>
-                Matches the <code>tool_call_id</code> of the last <code>role: "tool"</code>
-                message. Turn 1 has no tool message; turn 2 does.
+                Matches when the request&rsquo;s overall last message is a
+                <code>role: "tool"</code> result with that <code>tool_call_id</code>. Turn 1 has no
+                tool result last; turn 2 does.
+              </td>
+            </tr>
+            <tr>
+              <td>
+                Two tool-result outcomes that share a <code>tool_call_id</code> and differ only
+                inside the result payload (e.g. approve vs. cancel)
+              </td>
+              <td><code>toolResultContains</code></td>
+              <td>
+                Substring match on the last <code>role: "tool"</code> result&rsquo;s text. Use when
+                <code>toolCallId</code> alone can&rsquo;t tell the outcomes apart. Stateless.
               </td>
             </tr>
             <tr>
@@ -420,19 +445,25 @@ <h2 id="gotchas">Gotchas</h2>
             <strong>Substring vs. exact matching.</strong> Default matching is substring. Adding a
             <code>requestTransform</code> (e.g. to strip timestamps or request ids) flips matching
             to exact string equality &mdash; fixtures that previously matched as substrings will
-            silently stop matching. Only <code>userMessage</code> and <code>inputText</code> flip;
-            fields like <code>toolName</code> and <code>toolCallId</code> are always exact. Pin
-            exact strings in your fixtures when you use a transform.
+            silently stop matching. Only the <em>string</em> forms of <code>userMessage</code>,
+            <code>systemMessage</code>, and <code>inputText</code> flip; regex and
+            array-of-substrings forms are unaffected, and <code>toolResultContains</code> remains a
+            substring match even under a transform. Fields like <code>toolName</code> and
+            <code>toolCallId</code> are always exact. Pin exact strings in your fixtures when you
+            use a transform.
           </li>
           <li>
             <strong>Duplicate <code>userMessage</code> warnings.</strong>
             <code>validateFixtures</code> warns when two fixtures share the same
-            <code>userMessage</code> with identical <code>turnIndex</code>,
-            <code>hasToolResult</code>, and <code>sequenceIndex</code> values. Fixtures that differ
-            on any of these discriminators do not trigger the warning. Other fields like
-            <code>toolCallId</code>, <code>model</code>, and <code>predicate</code> are not factored
-            in, so the warning may still fire when those discriminators are present. Treat it as
-            advisory.
+            <code>userMessage</code> with identical values for <em>every</em> match discriminator
+            the router gates on &mdash; <code>userMessage</code>, <code>systemMessage</code>,
+            <code>inputText</code>, <code>toolCallId</code>, <code>toolResultContains</code>,
+            <code>toolName</code>, <code>model</code>, <code>responseFormat</code>,
+            <code>endpoint</code>, <code>context</code>, <code>sequenceIndex</code>,
+            <code>turnIndex</code>, and <code>hasToolResult</code>. Fixtures that differ on any one
+            of these discriminators do not trigger the warning. A <code>predicate</code> cannot be
+            compared for equivalence, so any fixture carrying one is keyed by its own identity and
+            is never reported as a duplicate. Treat it as advisory.
           </li>
           <li>
             <strong>First-wins ordering.</strong> Fixtures are evaluated in registration order (and,

docs/record-replay/index.html

@@ -1007,8 +1007,8 @@ <h2>Request Transform</h2>
         </div>
 
         <p>
-          When <code>requestTransform</code> is set, string matching for
-          <code>userMessage</code> and <code>inputText</code> switches from substring
+          When <code>requestTransform</code> is set, string matching for <code>userMessage</code>,
+          <code>systemMessage</code>, and <code>inputText</code> switches from substring
           (<code>includes</code>) to exact equality (<code>===</code>). This prevents shortened keys
           from accidentally matching unrelated prompts. Without a transform, the existing
           <code>includes</code> behavior is preserved for backward compatibility.

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@copilotkit/aimock",
-  "version": "1.36.1",
+  "version": "1.37.0",
   "description": "Mock infrastructure for AI application testing — LLM APIs, image generation, image editing, text-to-speech, transcription, audio translation, audio generation, video generation, embeddings, MCP tools, A2A agents, AG-UI event streams, vector databases, search, rerank, and moderation. One package, one port, zero dependencies.",
   "license": "MIT",
   "keywords": [
@@ -169,7 +169,7 @@
     "test:exports": "publint && attw --pack .",
     "lint": "eslint .",
     "format:check": "prettier --check .",
-    "release": "pnpm build && pnpm test && pnpm lint && npm publish",
+    "release": "pnpm format:check && pnpm build && pnpm test && pnpm lint && pnpm test:exports && npm publish",
     "prepare": "husky || true"
   },
   "lint-staged": {

packages/aimock-pytest/pyproject.toml

@@ -4,15 +4,15 @@ build-backend = "hatchling.build"
 
 [project]
 name = "aimock-pytest"
-version = "0.4.0"
+version = "0.5.0"
 description = "pytest fixtures for aimock — mock LLM APIs, multimedia, MCP, A2A, AG-UI, vector DBs, and more"
 readme = "README.md"
 requires-python = ">=3.10"
 license = "MIT"
 dependencies = ["requests>=2.28"]
 
 [project.optional-dependencies]
-test = ["pytest>=7.0", "requests>=2.28"]
+test = ["pytest>=7.0"]
 
 [project.entry-points."pytest11"]
 aimock = "aimock_pytest.plugin"