|
| 1 | +# scenecast |
| 2 | + |
| 3 | +> One asset definition. One typed JSON shape. Many rendering targets. The agent's view of the world, cast onto any surface. |
| 4 | +
|
| 5 | +Typed asset shapes + visual + headless **views** for AI agents. Authors return **WidgetData** JSON per size; the framework converts: |
| 6 | + |
| 7 | +- **JSON** (`WidgetData`) — the canonical primitive every author writes. Same JSON shape Daslab iOS, Daslab web, and any third-party renderer can consume. |
| 8 | +- **HTML** — rendered from WidgetData for humans, dashboards, the [scene-otel](https://github.com/daslabhq/scene-otel) scrubber, iOS/web viewers |
| 9 | +- **Markdown** — rendered from WidgetData for LLM context injection (3–5× cheaper in tokens than dumping raw JSON, while preserving the structure agents need to reason about) |
| 10 | +- **Text** — rendered from WidgetData for terminals and text-only models |
| 11 | +- **A2UI** — rendered from WidgetData as [A2UI v0.9](https://a2ui.org) envelope messages (`createSurface` + `updateComponents`) so the same scene can stream to any portable agent-UI client (Lit/React/Angular/Flutter/OpenClaw) |
| 12 | +- **MCP Apps** — rendered from WidgetData as an [MCP Apps](https://blog.modelcontextprotocol.io/posts/2026-01-26-mcp-apps/) tool resource (`text/html;profile=mcp-app`) for iframe-based hosts: Claude, ChatGPT, VS Code, Goose, Cursor |
| 13 | + |
| 14 | +[](https://a2ui.org/specification/v0.9-a2ui/) [](https://blog.modelcontextprotocol.io/posts/2026-01-26-mcp-apps/) [](./src/render/a2ui.test.ts) |
| 15 | + |
| 16 | +**Both 2026 agent-UI standards covered:** |
| 17 | +- *A2UI camp (declarative, native-rendered):* [`@a2ui/lit`](https://a2ui.org/reference/renderers/) · [`@a2ui/react`](https://a2ui.org/reference/renderers/) · [`@a2ui/angular`](https://a2ui.org/reference/renderers/) · [Flutter GenUI](https://docs.flutter.dev/ai/genui) · [OpenClaw Canvas](https://docs.openclaw.ai/platforms/mac/canvas) · [ADK Web](https://github.com/google/adk-web) · [CopilotKit / AG-UI](https://www.copilotkit.ai/) · [json-render](https://json-render.dev/docs/a2ui) |
| 18 | +- *MCP Apps camp (sandboxed iframe):* [Claude](https://blog.modelcontextprotocol.io/posts/2026-01-26-mcp-apps/) · [ChatGPT](https://developers.openai.com/apps-sdk/) · VS Code · Goose · Cursor |
| 19 | + |
| 20 | +A2UI output is validated against Google's `@a2ui/web_core/v0_9` Zod schemas in CI; MCP Apps bundles ship a `tools/call` postMessage bridge per the MCP Apps spec. |
| 21 | + |
| 22 | +Ships with **7 canonical types** in core — abstract primitives any vendor can implement: **Email**, **Message**, **Contact**, **Event**, **Task**, **Document**, **Mesh** (3D file in any format). Plus a library of widget primitives (Icon, Stack, List, Table, Metric, MetricGrid, KeyValue, Status, Document, Calendar, Plan, Empty, **Model3D**). |
| 23 | + |
| 24 | +The **Mesh** type is the spatial seam — domain extensions like `protein/structure`, `robot/arm`, `printable/object`, `lab/labware` all `extends: ["mesh/file"]` and inherit format-discriminated rendering (`<model-viewer>` for glb/gltf/usdz today; Mol*, urdf-loader, three-stl-loader to follow). Same multi-target pipeline as Email — agents reason about a 3D asset in the same compact Markdown summary they use for everything else. |
| 25 | + |
| 26 | +**Vendor extensions ship with benchmarks, not core.** Gmail, Slack, Salesforce, SAP S/4HANA — these live in **benchmark-scoped repos** like [`scenebench`](https://github.com/daslabhq/scenebench), which delivers scenecast extensions for every vendor in its benchmark domain. Each benchmark gets a magnet URL on `daslab.dev` (e.g. `daslab.dev/labs/automationbench`, `daslab.dev/s4bench`) showing its tasks, vendor schemas, and a leaderboard — scrubbable and scored. |
| 27 | + |
| 28 | +Vendor types declare `extends: ["email/mailbox"]` etc. — tools that consume canonical types work uniformly across all vendors that implement them. |
| 29 | + |
| 30 | +**[Live gallery →](https://daslabhq.github.io/scenecast/)** |
| 31 | + |
| 32 | +## Install |
| 33 | + |
| 34 | +```bash |
| 35 | +npm install scenecast |
| 36 | +``` |
| 37 | + |
| 38 | +## Use |
| 39 | + |
| 40 | +```ts |
| 41 | +import { Email } from "scenecast"; |
| 42 | + |
| 43 | +const inboxState = { messages: await fetchInbox() }; // any vendor — Gmail, Outlook, IMAP, … |
| 44 | + |
| 45 | +// Visual — drop into any HTML surface |
| 46 | +document.querySelector("#inbox")!.innerHTML = |
| 47 | + Email.defaultView.toHTML(inboxState, { size: "medium" }); |
| 48 | + |
| 49 | +// Headless — feed your agent compact, structured context |
| 50 | +const ctx = Email.defaultView.toMarkdown(inboxState, { size: "medium" }); |
| 51 | +// "Inbox (5 messages, 3 unread) |
| 52 | +// |
| 53 | +// - **Invoice #4421 — overdue** — from alice@vendor.com · unread |
| 54 | +// - **Quick question about Q2 plan** — from ceo@company.com · unread |
| 55 | +// - …" |
| 56 | + |
| 57 | +await llm.chat({ messages: [{ role: "user", content: ctx }, ...] }); |
| 58 | +``` |
| 59 | + |
| 60 | +### Render to A2UI for any portable agent-UI client |
| 61 | + |
| 62 | +```ts |
| 63 | +import { Email, render } from "scenecast"; |
| 64 | + |
| 65 | +const widget = Email.defaultView.toJSON(inboxState, { size: "medium" }); |
| 66 | +const messages = render.renderA2UI(widget, { surfaceId: "inbox-1" }); |
| 67 | +const wire = render.toA2UIJSONL(messages); |
| 68 | +// {"version":"v0.9","createSurface":{"surfaceId":"inbox-1","catalogId":"…/basic-catalog"}} |
| 69 | +// {"version":"v0.9","updateComponents":{"surfaceId":"inbox-1","components":[ |
| 70 | +// {"id":"root","component":"Column","children":["c2","c4"]}, |
| 71 | +// {"id":"c2","component":"Text","text":"Inbox","variant":"h2"}, |
| 72 | +// … |
| 73 | +// ]}} |
| 74 | + |
| 75 | +// stream `wire` to any A2UI v0.9 client — @a2ui/lit, @a2ui/react, |
| 76 | +// Flutter GenUI, OpenClaw Canvas, ADK Web, CopilotKit, … |
| 77 | +ws.send(wire); |
| 78 | +``` |
| 79 | + |
| 80 | +### Render to MCP Apps for Claude / ChatGPT / VS Code / Goose / Cursor |
| 81 | + |
| 82 | +```ts |
| 83 | +import { Email, render } from "scenecast"; |
| 84 | + |
| 85 | +const widget = Email.defaultView.toJSON(inboxState, { size: "medium" }); |
| 86 | +const resource = render.renderMCPApp(widget, { uri: "ui://inbox/widget" }); |
| 87 | +// { uri: "ui://inbox/widget", |
| 88 | +// mimeType: "text/html;profile=mcp-app", |
| 89 | +// text: "<!DOCTYPE html>…<script>…tools/call bridge…</script>" } |
| 90 | + |
| 91 | +// Return as a tool result resource — Claude, ChatGPT, VS Code et al. |
| 92 | +// will render it in a sandboxed iframe and bridge events via JSON-RPC |
| 93 | +// over postMessage on the ui/* namespace. |
| 94 | +return { content: [{ type: "resource", resource }] }; |
| 95 | +``` |
| 96 | + |
| 97 | +The same `WidgetData` flows to every renderer — your agent emits structure once and humans, terminals, LLMs, A2UI clients, and MCP-Apps hosts all consume from one source. |
| 98 | + |
| 99 | +### Addressability — every sub-element is anchorable |
| 100 | + |
| 101 | +Selections, annotations, deep links, agent tool targets, scene-otel spans — they all need to *address* a specific row, step, event, or 3D object inside a widget. The asset is the atom; addressing has to live on the atom. |
| 102 | + |
| 103 | +Every WidgetData sub-element with stable identity (`ListItem`, `PlanStep`, `CalendarEventEntry`, `MetricWidget`, …) carries an optional `id?: string`. Renderers emit `data-widget-anchor="<selector>"` on each rendered element using a small grammar: |
| 104 | + |
| 105 | +``` |
| 106 | +widget whole asset |
| 107 | +item[<id>] list / calendar item |
| 108 | +row[<index>] | row[<id>] table row |
| 109 | +field[<key>] keyed field (KeyValue, Status detail) |
| 110 | +step[<id>] plan step |
| 111 | +metric[<id>] metric in a metric_grid |
| 112 | +zone[<name>] floorplan zone |
| 113 | +object[<id>] spatial scene placed item |
| 114 | +surface[<id>] 3D mesh face / glTF node |
| 115 | +point[<x>,<y>] | point[<x>,<y>,<z>] raw 2D / 3D point |
| 116 | +``` |
| 117 | + |
| 118 | +```ts |
| 119 | +import { anchor, anchorRef, parseAnchor } from "scenecast"; |
| 120 | + |
| 121 | +anchorRef("inbox-1", anchor.item("m4")); |
| 122 | +// { asset_id: "inbox-1", anchor: "item[m4]" } |
| 123 | + |
| 124 | +parseAnchor("point[1.2,0.5,2.3]"); |
| 125 | +// { kind: "point", x: 1.2, y: 0.5, z: 2.3 } |
| 126 | +``` |
| 127 | + |
| 128 | +scenecast doesn't render annotations — that's a runtime concern (drawing arrows, badges, comment popovers requires geometry the type system doesn't have). What it owns is the **contract**: every consumer reads the same selectors, the same atoms, the same world model. |
| 129 | + |
| 130 | +## Why this exists |
| 131 | + |
| 132 | +Today most agents do one of two things with their world state, and both are bad: |
| 133 | + |
| 134 | +1. **Dump raw JSON into the context** — wastes tokens, hurts comprehension, makes long-running agents expensive |
| 135 | +2. **Hand-write a custom summarizer per app** — every team rebuilds Gmail-summarize, Salesforce-summarize, Stripe-summarize, … — none consistent, none shared |
| 136 | + |
| 137 | +scenecast gives you **one definition, five rendering targets, ten apps batteries-included.** Lazy users get good defaults. Power users override per view. |
| 138 | + |
| 139 | +## API |
| 140 | + |
| 141 | +### `defineAsset({ type, schema, defaultView, … })` |
| 142 | + |
| 143 | +```ts |
| 144 | +import { defineAsset, defineView } from "scenecast"; |
| 145 | + |
| 146 | +const Gmail = defineAsset({ |
| 147 | + type: "gmail/account", |
| 148 | + schema: gmailSchema, // JSON Schema for the asset's state |
| 149 | + defaultView: GmailInboxView, // see below |
| 150 | + views: { drafts: GmailDraftsView }, |
| 151 | + secretFields: ["access_token"], |
| 152 | + mockState: () => ({ messages: [...] }), // for tests + galleries |
| 153 | +}); |
| 154 | +``` |
| 155 | + |
| 156 | +### `defineView({ name, toHTML, toMarkdown, toText? })` |
| 157 | + |
| 158 | +```ts |
| 159 | +const GmailInboxView = defineView<GmailState>({ |
| 160 | + name: "GmailInbox", |
| 161 | + toHTML(state) { |
| 162 | + return `<div>… HTML …</div>`; |
| 163 | + }, |
| 164 | + toMarkdown(state) { |
| 165 | + const unread = state.messages.filter(m => !m.is_read).length; |
| 166 | + return `Inbox (${state.messages.length} msgs, ${unread} unread)\n\n` + |
| 167 | + state.messages.map(m => `- **${m.subject}** — from ${m.from_}`).join("\n"); |
| 168 | + }, |
| 169 | + // toText defaults to stripping HTML tags if not provided |
| 170 | +}); |
| 171 | +``` |
| 172 | + |
| 173 | +### View primitives |
| 174 | + |
| 175 | +Most asset views compose a small library of generic primitives: |
| 176 | + |
| 177 | +```ts |
| 178 | +import { primitives } from "scenecast"; |
| 179 | + |
| 180 | +const { TableView, MetricView, ListView, KeyValueView, |
| 181 | + CalendarView, StatusView, DocumentView, ImageView, PlanView } = primitives; |
| 182 | + |
| 183 | +TableView.toHTML({ |
| 184 | + title: "Open opportunities", |
| 185 | + columns: ["Name", "Amount", "Stage"], |
| 186 | + rows: [{ Name: "Meridian", Amount: "$245k", Stage: "Won" }, …], |
| 187 | +}); |
| 188 | +``` |
| 189 | + |
| 190 | +Each primitive ships HTML + Markdown out of the box. |
| 191 | + |
| 192 | +## Pairs with `scene-otel` |
| 193 | + |
| 194 | +When emitting trace events with [scene-otel](https://github.com/daslabhq/scene-otel), pass the asset directly — the schema becomes the type contract for the snapshot, and the default view powers the scrubber's rendering automatically: |
| 195 | + |
| 196 | +```ts |
| 197 | +import { Gmail } from "scenecast"; |
| 198 | +import { scene } from "scene-otel"; |
| 199 | + |
| 200 | +scene.set(Gmail.type, world.gmail); // schema-validated emit |
| 201 | +``` |
| 202 | + |
| 203 | +The scene-otel scrubber auto-detects registered scenecast assets and renders cards with the asset's default view. |
| 204 | + |
| 205 | +## Schemas |
| 206 | + |
| 207 | +The canonical `WidgetData` JSON Schema is published at [`schemas/scenecast.widgets.v0.json`](./schemas/scenecast.widgets.v0.json) — every widget kind (Table, Metric, List, …, Model3D) with its required + optional fields. Use it to validate views authored outside the TypeScript library. |
| 208 | + |
| 209 | +## Roadmap |
| 210 | + |
| 211 | +v0.1.0 (current) |
| 212 | + |
| 213 | +- ✅ `defineAsset` + `defineView` core |
| 214 | +- ✅ Multi-format render: HTML + Markdown + Text + A2UI v0.9 + MCP Apps |
| 215 | +- ✅ A2UI output validated against `@a2ui/web_core/v0_9` Zod schemas in CI |
| 216 | +- ✅ MCP Apps bundle ships the `tools/call` postMessage bridge for Claude / ChatGPT / VS Code / Goose / Cursor |
| 217 | +- ✅ Widget primitive library: Icon · Stack · List · Table · Metric · MetricGrid · KeyValue · Status · Document · Calendar · Plan · Empty · Model3D |
| 218 | +- ✅ 7 canonical types with mock state — Email, Message, Contact, Event, Task, Document, Mesh |
| 219 | +- ✅ Live gallery showing every asset side-by-side in HTML, Markdown, live A2UI (`@a2ui/lit`), and an MCP Apps sandboxed iframe — including a live `<model-viewer>` rendering of the Mesh asset |
| 220 | +- ✅ Anchor grammar — every sub-element is addressable (`item[<id>]`, `row[<idx>]`, `field[<key>]`, `step[<id>]`, `metric[<id>]`, `point[<x>,<y>,<z>]`, …) so consumers can bind selections, annotations, agent tool targets to specific rows / steps / events without reinventing addressing |
| 221 | + |
| 222 | +Coming next |
| 223 | + |
| 224 | +- **Native A2UI catalog** — publish a scenecast catalog so renderers render higher-level widgets (Plan, Status, Calendar) natively instead of falling back to basic-catalog primitives |
| 225 | +- **Incremental A2UI updates** — emit `updateComponents` patches per scene change instead of full snapshots |
| 226 | +- **A2UI action ingest** — wire client-side button taps back to tool invocations |
| 227 | +- **Image-format render** — Satori-based PNG rendering for vision-capable models |
| 228 | +- **Action handlers** — views declare `actions: { approve, redo, send }`; runtime routes scene action events |
| 229 | +- **`defineCheck`** — LLM-judged predicates as a sibling primitive (composable into milestones) |
| 230 | +- **AutomationBench bridge** — auto-translate AB's `assertions` to milestones; visualize first-unsatisfiable-step |
| 231 | +- **More assets** — HubSpot, Asana, Trello, Zoom, Linear, … |
| 232 | + |
| 233 | +## License |
| 234 | + |
| 235 | +MIT. See [LICENSE](./LICENSE). |
| 236 | + |
| 237 | +## Related |
| 238 | + |
| 239 | +- [`scene-otel`](https://github.com/daslabhq/scene-otel) — wire format for snapshotting agent state to OTel events. Pairs naturally — scenecast renders what scene-otel snapshots. |
| 240 | +- [`scenebench`](https://github.com/daslabhq/scenebench) — open harness for running, measuring, and visualizing agent benchmarks. Vendor types are authored with scenecast. |
| 241 | +- [`scenegrad`](https://github.com/daslabhq/scenegrad) — runtime goal assertions for agents. Scenecast schemas describe what is; scenegrad asserts what must be. |
| 242 | +- [`agent-otel`](https://github.com/mirkokiefer/agent-otel) — OTel router for agent telemetry. |
| 243 | +- [`autocompile`](https://github.com/mirkokiefer/autocompile) — observes repeated agent runs, compiles invariant parts to code. |
0 commit comments