feat: Pipeline run recording & deterministic replay - turn every run into a shareable, diffable, testable artifact

[Aarkin7]

Is your feature request related to a problem? Please describe.
Three problems that I keep hitting, and that I think most teams shipping Haystack to production run into too:

1. I can't reproduce LLM bugs. When a pipeline misbehaves in production (wrong route, hallucinated answer, a tool call that shouldn't have fired), the run is gone the moment it ends. The LLM is stochastic, retrievers depend on a live store, and web search returns different results every minute.
2. I can't write deterministic tests for my RAG and Agent pipelines. Every team I've talked to rebuilds the same hand-rolled mocking layer for CI: monkey-patch OpenAIChatGenerator, fake out SerperDevWebSearch, stub the retriever. The mocks drift from production behaviour within weeks, the tests rot, and we lose confidence in the suite. A quick grep for replay, recorder, cassette, or vcr across haystack/ and test/ returns nothing, there is no first-class primitive for this today.
3. I can't see what a run cost me. OpenAIChatGenerator writes usage into each ChatMessage._meta["usage"], but Pipeline.run() never aggregates it. If my pipeline has three generators and an Agent loop, I'm summing dictionaries by hand to get total tokens, and there's no pipeline.last_run_cost, no budget cap, no per-component breakdown. (This pain shows up in older discussions like Tracking LLM usage #9642 with no PR.)

These three look unrelated, but they're all the same shape: an executed pipeline run is ephemeral, there's no artifact you can save, share, replay, or measure.

Describe the solution you'd like
A new haystack.recording module that adds three things, built as one cohesive primitive:

from haystack.recording import load_run, diff_runs, ReplayMode

# 1. RECORD: opt-in flag, default behaviour unchanged
result, run = pipeline.run(data, record=True)

run.save("debug/run.json")
print(run.usage)          # {"prompt_tokens": 2104, "completion_tokens": 318, "total_tokens": 2422}
print(run.cost_estimate)  # {"openai/gpt-4o-mini": 0.0013, "total_usd": 0.0013}
print(run.timeline)       # per-component duration + exception (if any)

# 2. REPLAY: short-circuit side-effecting components (LLMs, web search, fetchers,
# remote retrievers) so the same pipeline runs deterministically with no network.
recorded = load_run("debug/run.json")
result = pipeline.run(data, replay=recorded, mode=ReplayMode.STRICT)
# Non-side-effecting components (builders, routers, joiners, validators) re-execute live
# so I can edit my prompt template and see how the run would have changed.

# 3. DIFF: A/B prompts, model swaps, retriever changes, with one call.
baseline = load_run("v1.json"); candidate = load_run("v2.json")
print(diff_runs(baseline, candidate))
# Structured diff: per-component IO deltas, decision-point changes, cost delta.

Implementation outline (designed to be additive, not invasive):

• New package haystack/recording/ modelled on haystack/tracing/, PipelineRun dataclass, PipelineRecorder context manager, Replayer, diff_runs.
• Hook the recorder into the same call site the breakpoint dispatch already lives in (haystack/core/pipeline/pipeline.py::_run_component), pre-call captures inputs, post-call captures outputs, duration, and any usage metadata. Agents extend the recording inside the existing loop in haystack/components/agents/agent.py.
• Pipeline.run() gains two keyword-only args: record=False and replay=None. Default behaviour is byte-identical to today.
• Components opt into "side-effecting" with a one-line class attribute (haystack_side_effecting = True), following the same dunder-attribute pattern Haystack already uses for component-level state markers like haystack_added_to_pipeline (see haystack/core/component/component.py:328). Built-in generators, web search, link fetchers, and audio transcribers ship with the flag in this PR; community components opt in when they want to.
• ReplayMode enum: STRICT (raise on drift), DRIFT_OK (re-run drifted components live), RECORD_NEW (extend the artifact in place).
• Serialization reuses existing Haystack utilities: PipelineSnapshot / PipelineState from haystack/dataclasses/breakpoints.py, _serialize_with_field_fallback from haystack/core/pipeline/breakpoint.py, and _serialize_value_with_schema from haystack/utils/base_serialization.py. The on-disk JSON is designed to embed the existing PipelineSnapshot schema so snapshot tooling can read records as a degenerate single-step snapshot.
• A SnapshotCallback-style hook (mirroring the existing one at haystack/core/pipeline/breakpoint.py:40) so users can stream records into S3, Datadog, or a database without touching disk.

Describe alternatives you've considered

• Stand-alone cost/token tracker: solves only problem no. 3 and leaves bug-repro + testing untouched. Half a feature.
• A standalone semantic LLM response cache: saves money but doesn't help reproduce a run or test it deterministically.
• Lean harder on PipelineSnapshot + breakpoints: snapshots exist, but they're a one-shot pause-and-save mechanism keyed to a manually-set Breakpoint. They don't capture per-component IO across an entire run, can't replay end-to-end without manual reconstruction, and don't help with cost or diffs.
• LangSmith / Phoenix / external diagnostics tools: these exist (some are paid SaaS, some are separate processes), but they sit outside the framework. Users have to pick, configure, and pay for them, and bug reports end up depending on third-party infra. A native primitive is portable, free, and shows up in every Haystack install by default.
• Pure mocking conventions in user test suites: what most teams do today. It works, until it doesn't: the mocks rot, drift from production behaviour, and offer zero help reproducing a real failed run.

None of the alternatives cover all three problems with one well-scoped abstraction.

Additional context
Why I think this belongs in Haystack core rather than as a third-party package:

• Haystack's README markets the framework as "a transparent architecture that lets you experiment, customize deeply, and deploy with confidence". Run recording is what makes that promise real, it turns the abstract "transparent and traceable" claim into a concrete JSON artifact a user can hold in their hand, share in an issue, and replay locally.
• Once this lands, bug reports against Haystack become "here's run.json, please replay." That's a categorically better debugging loop for the maintainers themselves, not just downstream users.
• Every Haystack-based app gets a deterministic test story for free, without API keys. That lowers the barrier to writing tests for community components and recipes, which is good for the ecosystem.
• No mainstream OSS LLM framework ships a built-in record/replay primitive today. LangChain users reach for LangSmith (paid SaaS); LlamaIndex users reach for Phoenix (separate process). A native, open primitive is a genuine OSS-first differentiator for Haystack.
• The breakpoint machinery, snapshot serialization, and per-component dispatch sites already exist, the change is mostly composition of existing primitives, not new infrastructure. Default behaviour stays byte-identical; this is purely additive.

I've done a fairly deep read of core/pipeline/pipeline.py, async_pipeline.py, breakpoint.py, components/agents/agent.py, and the generator components, and I'm confident this can be landed without breaking changes.

[Aarkin7]

I'd love to own the PR if the maintainers are open to it, happy to start with an RFC comment in this issue that nails down the JSON schema and the exact hook points before any code. Thanks for considering it!

[rautaditya2606]

Interesting proposal
i've been working on a related project, haystack diagnostics, that tackles the same underlying problem (ephemeral pipeline runs that are difficult to debug and reproduce) from a different angle

It doesnt implement record / replay, but instead focuses on document store validation, a retrieval failure taxonomy (NO_RESULTS, FILTER_EXCLUSION, SCORE_BELOW_CUTOFF, CONTEXT_LOSS, EMPTY_CONTEXT, GENERATOR_FAILURE), and debug bundle diffing (document rank/score deltas, configuration changes), which feels somewhat similar in spirit to diff_runs

theres no code overlap today, but if PipelineRun lands, I'd likely consume it as an input instead of maintaining my own debug bundle schema. sharing in case any of the failure taxonomy or diffing ideas are useful while you're designing this.

repo: https://github.com/rautaditya2606/haystack-diagnostics

[Aarkin7]

Thanks @rautaditya2606, really glad this surfaced early rather than after we both shipped incompatible formats.

Your retrieval failure taxonomy (NO_RESULTS, FILTER_EXCLUSION, SCORE_BELOW_CUTOFF, CONTEXT_LOSS, EMPTY_CONTEXT, GENERATOR_FAILURE) is a great fit for something a tool like yours could layer on top of PipelineRun. My instinct is to keep the recorder itself taxonomy-agnostic, just capture per-component IO faithfully (retriever inputs, top-k with scores, downstream generator output), so haystack-diagnostics can classify failures without extra instrumentation, and other tools can layer their own analyses too. One canonical artifact, many consumers.

I'll skim your repo this week and open a discussion on your side if I spot fields worth pre-reserving in the schema. Happy to iterate on the shape with you before it locks in.

[zeweihan]

This is the right primitive — and worth naming why it's load-bearing for regulated domains, not just a developer-experience nicety.

The three problems in the report feel unrelated but share one root in legal/clinical/financial corpora: an executed pipeline run is the only artifact that can answer "which combination of model + prompt + retrieved context + config produced this answer," and that question is a compliance requirement, not a debugging convenience. When a partner or regulator asks "why did the system flag (or miss) this clause," "the LLM is stochastic and the run is gone" is not a defensible answer. record=True producing a shareable, diffable artifact is exactly the audit trail those domains already mandate for non-AI systems (change-control sign-off, four-eyes review) — the gap is that AI pipelines have no equivalent today.

A few angles worth sharpening:

1. ReplayMode.STRICT raising on drift is the compliance-grade mode, and "drift" needs a precise definition. The most dangerous replay failure in regulated corpora isn't a loud component mismatch — it's a silent one: the retriever returns a slightly different chunk set (embedding store re-indexed overnight, a document was amended), the reranker reorders, and the replayed run produces a plausible-looking answer that differs from the recorded one with no exception raised. DRIFT_OK is fine for dev iteration, but for "does this recording still faithfully represent what production does?" you want drift to be visible, not absorbed. Concretely: the recording should capture enough of the retrieval inputs (not just the LLM I/O) that a STRICT replay can detect "the retrieved context for this query changed since recording" and surface it as a structured retrieval_drift finding rather than silently re-running live. Otherwise STRICT gives false confidence — it catches component-level divergence but misses the semantic drift that matters most for defensible outputs.

2. diff_runs is regulatory change-control for AI, and the unit of diff matters. The report frames diff as A/B prompt-and-model comparison, which is the dev framing. The regulated-domain framing is different: "we changed the retriever config on Tuesday — prove no answer that was correct on Monday degraded." That requires the diff to operate at decision-point granularity (which documents were retrieved, which tool fired, which route was taken), not just aggregate token/cost deltas. A diff that says "total cost went up 12%" is a billing artifact; a diff that says "for query Q, the retrieved-chunk set changed and the answer flipped from 'compliant' to 'flag'" is a change-control artifact. The RFC's decision-point changes line is the right instinct — worth making that the primary diff dimension, with cost/usage as secondary metadata.

3. Usage aggregation closes a real gap, but cost_estimate needs a provenance caveat. Aggregating per-component usage is genuinely missing today and valuable. One caveat: cost_estimate implies a precision the underlying pricing doesn't have (provider pricing changes, cached-input discounts, batch-vs-realtime rates). For a regulated-domain budget artifact that someone might attach to a compliance file, the word "estimate" should be loud and the pricing-source/timestamp should be in the recording so a reviewer knows which price list produced the number. A silently-wrong cost figure attached to an audit trail inherits the trust of the audit trail while being independently wrong — same plausible-but-wrong signature as retrieval drift.

4. The recording format is the long-term asset — design it to outlive component churn. The strongest parallel is test cassettes (VCR/httpretty): the value compounds when a recording from six months ago still loads and diffs against today's run. That requires the recording to capture component-agnostic I/O contracts (inputs/outputs as they cross the component boundary) rather than internal component state, so a generator swap (GPT-4o → Claude) still produces a meaningful diff rather than a format-mismatch error. The __haystack_side_effecting__ opt-in is the right granularity for which components to short-circuit, but the recording itself should be component-name-and-version annotated, not component-format-coupled.

This RFC threads a needle that most RAG frameworks don't attempt: making a single pipeline run into a first-class, durable, diffable object. For document-heavy domains where "which version produced this output" is a regulatory question, that's the gap between a demo and a deployable system.

[Aarkin7]

Thanks @zeweihan! The compliance-artifact framing is a stronger positioning than the DX one I opened with, and I'm going to fold it into the RFC.

Two of your points are load-bearing enough that I'll bake them into the schema from day one rather than treating them as later polish:

• Retrieval-drift as a first-class STRICT finding. You're right that catching only component-level mismatch gives false confidence. The recording will capture retriever inputs (query, filters, top-k, embedding-model fingerprint) alongside outputs, and STRICT will surface a structured retrieval_drift finding when the retrieved set diverges on replay instead of silently absorbing it.
• Decision-point-primary diff_runs. Which docs were retrieved, which route was taken, which tool fired, that becomes the primary diff dimension. Cost/usage drops to secondary metadata where it belongs.

On the other two: cost_estimate will carry an explicit pricing_source + pricing_snapshot_date so any downstream audit knows which price list produced the number, and the recording will be component-name-and-version annotated so a OpenAIChatGenerator → AnthropicChatGenerator swap still diffs meaningfully instead of failing on format mismatch.

Really appreciate the depth here.