feat(js/testing): mockModel + echoModel for genkit/testing#5475
Merged
Conversation
github-actions
Bot
added
docs
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces deterministic testing utilities for Genkit, specifically mockModel and echoModel, along with a sample application demonstrating their usage with both node:test and vitest. The review feedback highlights several key robustness and usability improvements in mockModel: handling potential DataCloneErrors when cloning requests with non-serializable properties, joining multi-turn messages with newlines in renderRequestText for better readability, and adding defensive checks in toResponseData to prevent runtime errors when a mock response is falsy.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
cabljac
force-pushed
the
feat/js-testing-module
branch
from
July 1, 2026 10:39
69370e7 to
61dc948
Compare
Promote Genkit's internal model-mocking helpers into the public
`genkit/testing` surface so app developers can unit-test flows,
prompts, tools, and chat deterministically — without a live model,
network, or API key.
- mockModel(ai, { respond }): a programmable mock model that drives each
call's response (text, structured, tool requests, streamed chunks) and
records calls with typed inspection (lastRequest / requests /
requestCount), replacing the untyped `(model as any).__test__*` hack.
- echoModel(ai): a zero-config model that echoes the rendered request as
text, for asserting prompt/message assembly.
- Re-exported from `@genkit-ai/ai/testing` and `genkit/testing`.
- Tests at js/genkit/tests/mock-model_test.ts.
- Sample app js/testapps/testing-sample demonstrating flow, tool,
streaming, and prompt-assembly tests.
Implements the JS proposal in docs/js-testing-module-rfc.md (RFC #20).
autoModel (zero-config schema-aware tier) is left for a follow-up per
RFC open question 5.
…view, fix sample scripts
- echoModel: render non-text parts (media, tool requests/responses, reasoning, data) as labelled placeholders instead of silently dropping them, so the echo reflects the full request the model would have seen. - mockModel: snapshot requests with structuredClone instead of JSON.parse(JSON.stringify(...)) — preserves more types and avoids silently stripping undefined fields. - mockModel: forward only the ModelInfo fields defineModel accepts (versions/label/supports) instead of spreading via `as any`. - testing-sample: expose a createMenuApp() factory and build a fresh, isolated app per test so mocks no longer re-register the default model name on a shared registry (removes registry-overwrite log spam). - bump copyright headers on new files to 2026.
Fresh-context review caught that renderPart still dropped two Part variants (ResourcePart, CustomPart) to empty string — the same silent-drop class the prior commit fixed for media/tool/reasoning/data.
The previous sample exercised mostly framework behavior because the flow was a passthrough. Reworked it around an app with logic worth testing: - recommendPrompt: a dotprompt (Handlebars template + system + tool), asserted via echoModel (system + rendered template). - recommendDish: a flow that requests STRUCTURED output, validates it, and derives `withinBudget` itself — tests pin down that derivation. The same model output with a lower budget flips the result, proving the test exercises app logic, not the model. Adds an error-path test. - streamRecommendation: a streaming flow driven via flow.stream(), so the streaming test goes through the flow instead of bypassing it. Structured output schema is supplied at the flow's call site (not baked into the prompt) so the prompt stays text-renderable for echoModel, which a strict output schema would reject. README updated. node:test 6/6, vitest 5/5, tsc clean.
…stText echoModel returns text, which can't satisfy a structured output schema — Genkit derives `output` by parsing the response text and validating it. Previously that surfaced as a cryptic ajv "must have required property" error. Now: - echoModel declares native constrained support, so the framework hands the schema to the model in `request.output.schema` (instead of injecting it as prompt text). echoModel detects it and throws an explanatory error pointing to the right pattern. - Add `model.lastRequestText` to every mock: the full assembled request (system + all messages) flattened to a string. This gives echo-style prompt-assembly assertions on ANY mock — including structured-output paths where echoModel can't be used — as pure request inspection, decoupled from the response shape. - Share the request-flattening logic (renderRequestText) between echoModel and lastRequestText; reframe echoModel's jsdoc as a text-path preset. Tests: echoModel throw-under-schema, lastRequestText on a structured path. Sample gains a lastRequestText assertion on the structured flow. README updated. genkit 9/9, sample node 7/7 + vitest 5/5, tsc clean.
Address review nits on the mock-model helpers: - renderRequestText joins messages with a newline so adjacent messages' text can't fuse into one token and silently break boundary-spanning assertions. - lastRequest and requests return structuredClone snapshots so callers can't mutate recorded history through a view. - toResponseData no longer lets an explicit `finishReason: undefined` clobber the 'stop' default. Each fix covered by a new test in mock-model_test.ts.
…ockModel Prove the merged mockModel already tests the two features most likely to be assumed to need dedicated mocks: - interrupt round-trip: a human-in-the-loop tool interrupts, generation pauses (response.interrupts), then resumes to completion via the real resume + tool.respond API. - agent handoff: a prompt-as-tool request swaps the system preamble, observable through model.lastRequestText. No new helpers required — an interrupt is a tool request the framework turns into a pause, and a handoff is a prompt-as-tool request.
… injection Borrow the highest-value ergonomics from AI SDK / LangChain test fakes: - respond now also accepts an array: a queue consumed one item per call with the last repeating, for scripted multi-turn tests without a branching callback. - a queued Error is thrown when reached, giving declarative failure injection (retry/fallback paths) without hand-throwing inside respond. - model.toolResponses exposes the tool results fed back to the model, so tests assert which tools ran and what they returned instead of digging through message content. The sample now uses it. Callback form is unchanged and still required for streaming.
…and interrupts in the sample Extend testing-sample so the sample app exercises the features added during review, not just the base mockModel/echoModel: - a confirmBooking interrupt tool shows a human-in-the-loop pause/resume round-trip (via restart). The app now imports genkit/beta since interrupts are a beta feature; everything else is unchanged. - response-queue scripting (respond: [...]) for a concise two-turn tool interaction with no branching callback. - error injection via a queued Error. - model.toolResponses to assert which tools ran and what they returned, replacing manual message digging. Covered under both node:test (10) and vitest (6); README updated.
…esults Address review on mockModel: - cloneRequest() falls back to a message/part-level copy when a request carries non-serializable values (e.g. a function in config), which structuredClone would otherwise reject with DataCloneError. - toResponseData treats a falsy respond result (a callback that streams but returns nothing) as an empty model response instead of throwing. Adds tests for both, and applies repo prettier formatting to the testing module and sample.
Regenerate js/pnpm-lock.yaml so pnpm i --frozen-lockfile succeeds in CI after the new testapp declared its dependencies.
cabljac
force-pushed
the
feat/js-testing-module
branch
from
July 1, 2026 11:44
60952e6 to
3bd9e54
Compare
mockModel now defaults supports.constrained to 'all', so a generate with
an output schema reaches respond with request.output.schema intact rather
than the framework injecting a schema blob into the prompt and stripping
it. This removes friction when mocking structured-output paths (including
in the Dev UI) and makes the mock resemble a modern provider model.
The default is spread last, so callers opt out with
supports: { constrained: 'none' } to exercise the simulated path.
echoModel is unchanged (it still forces 'all').
The beta Chat API (app.chat) and its cross-agent preamble swap were removed upstream in #5248, so the handoff test's app.chat(promptAgent) call threw "app.chat is not a function" after rebasing onto main. Rewrite it as a single-agent defineAgent(...).chat().send(...) tool loop, keeping the same mockModel surface under test: a multi-turn tool round trip with the agent's rendered system preamble asserted via lastRequestText and requestCount.
…odel Mock lifecycle was tied to registry registration: fresh behavior per test meant a fresh registration, pushing tests toward a fresh-Genkit-instance-per-test factory that diverges from the standard module-singleton app structure. Instead, follow the established mocking idiom (jest mocks, AI SDK / LangChain test fakes): register once, then - reset(): clears recorded history and re-arms the construction-time respond (a queue restarts from its first item) - respondWith(...): swaps respond behavior for subsequent calls - respond now also accepts a single static MockResponse, returned on every call Unit tests cover single-response respond, respondWith swap semantics, and reset restoring construction behavior after an override.
…-once mocks The sample previously modeled a createMenuApp() factory so each test could build an isolated registry. That prescribed a nonstandard app structure just to be testable. Now the app is a plain module-level singleton (the structure every other doc and sample uses) and tests register one mock per file, relying on the runner's per-file process isolation (node --test, jest, and vitest all provide it), with reset() in beforeEach and per-test respondWith(...). echoModel moves to its own test files (prompt_test.ts, prompt.vitest.test.ts) since menu tests already claim 'menuModel'.
node --test spawns a child process per file; Jest and Vitest isolate each file's module registry. 'Fresh process' overclaimed the latter.
cabljac
added a commit
to genkit-ai/docsite
that referenced
this pull request
Jul 2, 2026
Reworked to match the updated genkit-ai/genkit#5475 API: apps keep the standard module-level singleton, tests register one mock per file (runners isolate files in their own process/module graph) and use reset() in beforeEach with per-test respondWith(...). The fresh-instance-per-test factory is demoted to a short 'isolating tests further' note.
cabljac
marked this pull request as ready for review
July 15, 2026 06:57
Member
|
One thing worth deciding before merge: the |
Wires the testing-sample node:test and vitest suites into CI so mockModel/echoModel regressions fail the build, mirroring the esm testapp pattern.
cabljac
enabled auto-merge (squash)
July 15, 2026 16:27
pavelgj
approved these changes
Jul 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implementation of the JS testing-module RFC (#5438).
What
Promotes Genkit internal model-mocking helpers into the public
genkit/testingsurface so app developers can unit-test flows, prompts, tools, and chat deterministically, with no live model, network, or API key.mockModel(ai, { respond })- programmable mock.respond(req, { sendChunk })drives each call response, including text, structured output, tool requests, or a stream. Typed inspection includesmodel.lastMessage,model.lastRequest,model.requests, andmodel.requestCount.echoModel(ai)- zero-config model that echoes the rendered request as text, for asserting prompt/message assembly.@genkit-ai/ai/testingandgenkit/testing, alongside the existingtestModels.Inspection reads in Genkit idiom:
model.lastMessage!.textinstead of reaching into raw request message content.Files
js/ai/src/testing/mock-model.tsjs/ai/src/testing/index.ts,js/genkit/src/testing.tsjs/genkit/tests/mock-model_test.tsjs/testapps/testing-sample/recommendDishflow anddailySpecialtooljs/testapps/testing-sample/tests/menu_test.tsnode:testcoveragejs/testapps/testing-sample/tests/menu.vitest.test.tsWhy a dedicated fake model, not middleware
The RFC alternatives section weighs model middleware as the closest mechanism. This PR implements
mockModelandechoModelas realdefineModelactions using apiVersion v2, mirroring the existing internaldefineProgrammableModelapproach.Function.lengthstreaming behavior in model middleware dispatch.Scope notes
autoModelis not included; it remains an RFC open question for follow-up.Refs #5438.
Updates since first draft
Inspection members were renamed and expanded during review, and a few ergonomics were borrowed from the AI SDK / LangChain test fakes:
lastMessage→lastRequestMessage; addedlastRequestText(whole assembled conversation flattened, works even on structured-output paths whereechoModelcan't) andtoolResponses(tool results fed back to the model, so tests assert which tools ran and what they returned without digging through message content).respondalso accepts an array — a queue consumed one item per call, last item repeating — for scripted multi-turn tests without a branching callback.Erroris thrown when reached, giving declarative failure testing (retry/fallback paths).echoModeloutput-schema guard: throws a clear error if the request carries an output schema (prose can't satisfy it), pointing atmockModel+lastRequestTextinstead.structuredCloned and read-through views are cloned so tests can't mutate recorded history;renderRequestTextnewline-separates messages so adjacent text can't fuse.Tests grew accordingly:
mock-model_test.tsnow 17 pass (adds queue, error injection,toolResponses, defensive-copy, and — notably — interrupt round-trip and agent-handoff cases provingmockModelalready covers those without dedicated helpers). Sample: 7 node + 5 vitest.Update: resettable mocks (register-once idiom)
Mock lifecycle was tied to registry registration, which pushed tests toward a fresh-Genkit-instance-per-test factory - a nonstandard app structure prescribed just to be testable. Following the established fake-model idiom (jest mocks, AI SDK / LangChain test fakes) instead:
model.reset()- clears recorded history and re-arms the construction-timerespond(a queue restarts from its first item). Call inbeforeEach.model.respondWith(...)- swaps respond behavior for subsequent calls; recorded history untouched.respondaccepts a single static response, returned on every call (alongside the existing callback and queue forms).The testing-sample app is back to a standard module-level singleton. Tests register one mock per file under the app's default model name and rely on the runner's per-file isolation (
node --testspawns a child process per file; Jest and Vitest isolate each file's module graph), withreset()inbeforeEachand per-testrespondWith(...).echoModelmoved to its own test files since the menu tests claim the same model name.Tests: 25 unit (adds single-response,
respondWithswap,resetre-arm, and empty-string-response regression) + 10 node + 6 vitest.Follow-up (separate PR)
Chat/session testing has one small gap: there is no public in-memory
SessionStore, so users wanting to pre-seed / inspect / test their own store re-implement it (the internal tests literally do —TestMemorySessionStore). A production-qualityinMemorySessionStore()factory already exists atjs/ai/src/session.tsbut isn't exported. Follow-up: exportinMemorySessionStore()from the public package (optionally re-exported viagenkit/testing). This is a core public-API change, kept out of this testing-utility PR deliberately. Interrupts and agent handoff need no new helpers (covered by tests here);mockRetriever/mockEmbedderare out of scope pending their roadmap.