Add mastra auto-instrumentation by Qard · Pull Request #1901 · braintrustdata/braintrust-sdk-javascript

Stephen Belanger (Qard) · 2026-04-23T22:34:26Z

Fixes #1890

Summary

add Mastra auto-instrumentation configs and wire them into the hook, bundler plugin, webpack loader, and public exports
add Mastra tracing channels and a runtime plugin for agent, tool, workflow run, and workflow step spans with current-span context binding
add focused Mastra tests and an e2e scenario covering agent generate/stream, Tool.execute, workflow runs, and workflow-step tool nesting

Testing

pnpm --dir js exec vitest run src/auto-instrumentations/configs/mastra.test.ts src/instrumentation/plugins/mastra-plugin.test.ts src/instrumentation/registry.test.ts
pnpm run test:e2e -- mastra-instrumentation

Notes

pnpm --dir js test -- mastra and pnpm --dir js test -- auto-instrumentations still hit unrelated existing network-dependent failures in src/framework.test.ts and src/logger-misc.test.ts
pnpm run formatting still reports pre-existing unformatted fixture files under js/tests/auto-instrumentations/fixtures/test-files/

Luca Forstner (lforst)

Can we also add the mastra e2e test scenario to the ci summary?

Luca Forstner (lforst)

In the e2e test summary it looks like we don't capture LLM spans.

Do you think that is an issue?

…a directly to interface Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… unused types - Remove mastra from BraintrustPluginConfig inline type (api-compat flags any change to the inline type string; use getIntegrationConfig dynamic lookup instead, same pattern as groq in other integration PRs) - Remove unused IntegrationName/IntegrationMap type aliases (no longer referenced after widening getDefaultConfig/readEnvConfig return types) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…ases Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add internal RuntimeInstrumentationConfig and RuntimeBraintrustPluginConfig that extend the public types with the mastra integration flag. Users still opt out via BRAINTRUST_DISABLE_INSTRUMENTATION=mastra; the public interfaces are unchanged so api-compatibility passes without a major version bump. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Luca Forstner (lforst) · 2026-05-21T08:17:55Z

+function normalizeIntegrationName(name: string): IntegrationName {
+  switch (name) {
+    case "cursor-sdk":
+      return "cursorSDK";
+    case "githubcopilot":
+    case "github-copilot":
+    case "copilot-sdk":
+      return "gitHubCopilot";
+    case "openai-codex-sdk":
+      return "openaiCodexSDK";
+    case "@mastra/core":
+    case "mastra-core":
+      return "mastra";
+    default:
+      return name as IntegrationName;
+  }
+}


Did you find this necessary? It randomly widens our API scope and what we need to document.

Dropped the @mastra/core and mastra-core aliases in e9e3224 — mastra is the canonical key. The other aliases in normalizeIntegrationName (cursor-sdk → cursorSDK, github-copilot → gitHubCopilot, etc.) bridge case mismatches between env var conventions and the property names; mastra already matches both, so the extras were just noise. normalizeIntegrationName and IntegrationName are both module-local, not exported, so the public API surface is unchanged either way.

Luca Forstner (lforst) · 2026-05-21T08:23:02Z

@@ -0,0 +1,286 @@
+span_tree:


I don't see any type: llm spans here which is a bit worrying.

Fixed in e9e3224 — Mastra Workflow Step workflow.executionWorkflow.step.llm-execution is now typed as [llm] (it's the synthetic step where the model call actually happens). The new mastra-v1260.span-tree.txt shows two [llm] spans (one per agent.generate/stream invocation).

Can we loosen the normalization a bit on the scenario so we can see input & output of the llm spans?

Luca Forstner (lforst) · 2026-05-21T08:27:37Z

+const tracePromiseTransform =
+  (
+    codeTransformerTransforms as {
+      default?: {
+        tracePromise?: CustomTransform;
+      };
+      tracePromise?: CustomTransform;
+    }
+  ).tracePromise ??
+  (
+    codeTransformerTransforms as {
+      default?: {
+        tracePromise?: CustomTransform;
+      };
+    }
+  ).default?.tracePromise;


Can we add a comment explaining why this is necessary?

Added a header doc-comment in e9e3224 explaining the arrow-in-constructor problem the transform is solving.

Luca Forstner (lforst) · 2026-05-21T08:33:03Z

+const stableVersionRange = "1.26.0";
+const alphaVersionRange = "1.26.1-alpha.0";
+
+const stableAgentWorkflowChunks = [
+  { filePath: "dist/chunk-CXW3Z2OL.js", moduleType: "esm" },
+  { filePath: "dist/chunk-PBGNXXU5.cjs", moduleType: "cjs" },
+] as const;
+
+const alphaAgentWorkflowChunks = [
+  { filePath: "dist/chunk-4QTY73BW.js", moduleType: "esm" },
+  { filePath: "dist/chunk-7432OHPH.cjs", moduleType: "cjs" },
+] as const;
+
+const toolChunks = [
+  { filePath: "dist/chunk-O3JJ5ZPY.js", moduleType: "esm" },
+  { filePath: "dist/chunk-U7Z7GCXY.cjs", moduleType: "cjs" },
+] as const;
+


Are we really only instrumenting one specific version of the package?

I also have the gut feeling that the chunk hashes have a pretty high chance of updating between versions putting burden on us to update everything, and also our users being blocked from upgrading if they want to have tracing. While implementing, did you explore any other options of doing things? Maybe patching the new Mastra() constructor to pass in an observability object of some sort? https://mastra.ai/docs/observability/tracing/exporters/braintrust#zero-config-setup

You're right that the chunk-hash approach is the wrong long-term answer — every Mastra release will produce new chunk filenames, so we'll be playing catch-up. I restructured the config in e9e3224 into a mastraVersions table that makes adding a new release mechanical (install, grep dist/ for the agent/workflow/tool chunks, append an entry).

For the doc'd new Mastra({ observability: ... }) exporter pattern: yes, that's the right design and I want to switch to it, but it's a meaningful refactor — we need an orchestrion match for the Mastra constructor (or a runtime-side patch in hook.mjs that wraps the imported class), plus a Braintrust exporter that conforms to Mastra's exporter interface. I'd rather land this PR with the per-version mapping documented and ship constructor patching as a focused follow-up than block on the redesign.

Versions outside the current mastraVersions entries simply no-op (no spans) — the user's code still runs fine — so the graceful-degradation cost is acceptable in the interim.

I think going with the current approach and merging this PR doesn't really make sense in its current state. This PR is about adding mastra auto instrumentation and if it only works for a single version (which isn't even the latest or most downloaded version) we are missing the goal no?

I think we should immediately try to find a good solution here.

stale

- Mark workflow.executionWorkflow.step.llm-execution spans as [llm] type so the model call surfaces in the span tree - Type mastra channel args (agent self -> MastraAgentLike, etc.) - Drop the redundant `@mastra/core` / `mastra-core` env-var aliases - Generalize the code-transformer error message (not Mastra specific) - Add header comment to custom-transforms.ts explaining the arrow-in-constructor problem - Restructure mastra config into a `mastraVersions` table to make adding new Mastra releases mechanical, with a comment explaining the chunk-hash limitation and pointing toward constructor patching Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The single-cast form fails strict TS because the generic union type doesn't sufficiently overlap. Keep the cast but pin it to the precise target shape so the unsafety is contained. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…iterals (#2036) ## Summary - `areInterfaceSignaturesCompatible` was flagging adding an optional property to an inline object literal type (e.g. `mastra?: boolean` on `InstrumentationConfig.integrations`) as a breaking change. - Root cause: it compared each property's *type* as a string, then only re-checked via `isUnionTypeWidening` when the strings differed. Inline object literals aren't unions, so any new field tipped it into "modified" territory. - Fix: delegate to the existing `areObjectTypeDefinitionsCompatible` helper when both sides are inline object literals, mirroring the pattern already in `areTypeAliasSignaturesCompatible`. Also apply the same delegation inside `areObjectTypeDefinitionsCompatible` itself so nested object literals are handled the same way. - Lifted the inline `isObjectType` helper to a module-scope `isObjectLiteralType` so the interface, object-literal, and intersection comparators share one definition. After this lands, the `js-api-compatibility (20)` check on PRs that add optional integration keys (#1891, #1897, #1901, future SDK integrations) will pass instead of failing informationally. ## Test plan - [x] Added 4 regression tests in `describe("areInterfaceSignaturesCompatible", ...)`: - optional property added to inline object literal (the `InstrumentationConfig` case) → passes - required property added → still rejected - property removed → still rejected - optional property added to deeply nested inline object literal → passes - [x] `pnpm exec vitest run tests/api-compatibility/api-compatibility.test.ts` — all 14 interface-compat tests pass plus the rest of the file - [x] `pnpm run fix:formatting` and `pnpm run lint` clean #skip-changeset — this is a test-only change with no impact on the published package. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Stephen Belanger (Qard) requested review from Abhijeet Prasad (AbhiPrasad) and Luca Forstner (lforst) April 23, 2026 22:34

Stephen Belanger (Qard) self-assigned this Apr 23, 2026

Luca Forstner (lforst) previously approved these changes Apr 24, 2026

View reviewed changes

Stephen Belanger (Qard) force-pushed the mastra-auto-instrumentation branch from 37f8aaa to 43c826c Compare May 6, 2026 17:11

Stephen Belanger (Qard) requested a review from Luca Forstner (lforst) May 6, 2026 20:07

Stephen Belanger (Qard) force-pushed the mastra-auto-instrumentation branch from 443f70c to 8d58469 Compare May 6, 2026 20:13

Luca Forstner (lforst) reviewed May 6, 2026

View reviewed changes

Stephen Belanger (Qard) force-pushed the mastra-auto-instrumentation branch from c01e4fd to ffcc4ea Compare May 7, 2026 16:01

Stephen Belanger (Qard) force-pushed the mastra-auto-instrumentation branch 2 times, most recently from 93440d7 to c21b696 Compare May 20, 2026 09:31

Stephen Belanger (Qard) and others added 10 commits May 21, 2026 15:20

Add mastra auto-instrumentation

53c34c4

Fix Mastra CI checks

ac882b0

Fix Mastra CI follow-ups

a203d0b

Fix Mastra canary e2e snapshots

0798115

Add Mastra to e2e CI summary

3b3cac4

fix(api-compat): restore groq to BraintrustPluginConfig and add mastr…

59fd430

…a directly to interface Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

fix(instrumentation): restore IntegrationName/IntegrationMap type ali…

b3a7e51

…ases Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

chore: bump mastra changeset to minor for public interface addition

c96aea4

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Stephen Belanger (Qard) force-pushed the mastra-auto-instrumentation branch from b59c93e to 282b413 Compare May 21, 2026 07:21

Add spantree snapshots

542caed

Luca Forstner (lforst) reviewed May 21, 2026

View reviewed changes

Stephen Belanger (Qard) and others added 3 commits May 21, 2026 18:15

Drop registry test for removed @mastra/core/mastra-core env aliases

83b8cc1

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Stephen Belanger (Qard) mentioned this pull request May 21, 2026

fix(api-compat): allow optional property additions to inline object literals #2036

Merged

3 tasks

Conversation

Stephen Belanger (Qard) commented Apr 23, 2026

Summary

Testing

Notes

Uh oh!

Luca Forstner (lforst) left a comment

Choose a reason for hiding this comment

Uh oh!

Luca Forstner (lforst) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants