Skip to content

Commit ed79578

Browse files
mirkokiefermirkokiefer
committed
feat(spatial): mesh/file canonical + Model3DWidget + live model-viewer
The first spatial primitive — one canonical asset, format-discriminated rendering. Same multi-target pipeline (HTML / Markdown / Text / A2UI / MCP Apps) Email and Document already use; the format hint picks the viewer (model-viewer for glb/gltf/usdz today; Mol*, urdf-loader, three-stl-loader to follow). - New Model3DWidget {uri, format, name?, bounds?, vertexCount?, posterUri?, cameraOrbit?, autoRotate?} added to the WidgetData union; every renderer extended with a model_3d case. - Mesh canonical type at "mesh/file" — schema + view + mockState that ships KhronosGroup's Damaged Helmet glTF (CC-BY-4.0). Domain extensions (protein/structure, robot/arm, printable/object) will declare extends: ["mesh/file"]. - Gallery loads <model-viewer> once at page level; MCP Apps bundle conditionally injects the script when the rendered HTML contains a <model-viewer> element so the same bundle renders 3D inside Claude / ChatGPT / VS Code / Goose / Cursor sandboxes too. - Markdown summary stays compact for LLM context: "GLB · 14,556 verts · 1.70m × 1.70m × 1.70m" — agents reason about 3D state in the same token-cheap projection they already use for everything else. This is the seam the spatial coordinate-system thesis hangs on. Same agent runtime, same approval gates, same OTel trace — now also for 3D files. Protein structures, URDF robots, STL printables all stack on top via extends.
1 parent 6d6f31c commit ed79578

11 files changed

Lines changed: 1476 additions & 5 deletions

File tree

‎gallery/build.ts‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -463,6 +463,40 @@ main { max-width: 1400px; margin: 0 auto; padding: 32px 24px 80px; }
463463
.ws-img img { max-width: 100%; border-radius: 6px; border: 1px solid var(--border); }
464464
.ws-img figcaption { color: var(--muted); font-size: 12px; margin-top: 6px; }
465465
466+
/* ----- 3D model widget ----- */
467+
.ws-model3d {
468+
display: flex; flex-direction: column; gap: 6px;
469+
width: 100%; height: 100%;
470+
min-height: 200px;
471+
}
472+
.ws-model3d model-viewer {
473+
flex: 1; min-height: 200px;
474+
width: 100%;
475+
background: #f1f5f9;
476+
border-radius: 8px;
477+
}
478+
.ws-model3d-meta {
479+
font-size: 11px; color: var(--muted); text-align: center;
480+
font-family: ui-monospace, "SF Mono", Menlo, monospace;
481+
}
482+
.ws-model3d-fallback {
483+
display: grid; grid-template-columns: 64px 1fr; gap: 12px;
484+
padding: 12px; background: var(--bg);
485+
border: 1px solid var(--border); border-radius: 8px;
486+
}
487+
.ws-model3d-poster {
488+
width: 64px; height: 64px;
489+
display: flex; align-items: center; justify-content: center;
490+
background: #e2e8f0; border-radius: 8px;
491+
font-family: ui-monospace, "SF Mono", Menlo, monospace;
492+
font-size: 11px; font-weight: 700; color: #475569;
493+
letter-spacing: 0.04em;
494+
}
495+
.ws-model3d-poster img { width: 100%; height: 100%; object-fit: cover; border-radius: 8px; }
496+
.ws-model3d-name { font-size: 14px; font-weight: 600; }
497+
.ws-model3d-link { font-size: 11px; color: var(--accent); text-decoration: none; margin-top: 4px; display: inline-block; }
498+
.ws-model3d-link:hover { text-decoration: underline; }
499+
466500
.ws-plan-row { display: grid; grid-template-columns: 24px 1fr; gap: 8px; padding: 4px 0; }
467501
.ws-plan-mark { color: var(--muted); font-size: 16px; }
468502
.ws-plan-completed .ws-plan-mark { color: var(--green); }
@@ -581,6 +615,9 @@ const html = `<!DOCTYPE html>
581615
<meta charset="UTF-8">
582616
<meta name="viewport" content="width=device-width,initial-scale=1">
583617
<title>scenecast · view gallery</title>
618+
<!-- Google's <model-viewer> for glb/gltf/usdz cells. Loaded once at the
619+
page level — multiple ES module imports of the same URL are deduped. -->
620+
<script type="module" src="https://ajax.googleapis.com/ajax/libs/model-viewer/4.0.0/model-viewer.min.js"></script>
584621
<style>${css}</style>
585622
</head>
586623
<body>

‎gallery/index.html‎

Lines changed: 1083 additions & 2 deletions
Large diffs are not rendered by default.

‎src/index.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ export type {
4848
MetricWidget, MetricGridWidget, KeyValueWidget,
4949
StatusWidget, DocumentWidget, CalendarWidget,
5050
PlanWidget, EmptyWidget,
51+
Model3DWidget, Model3DFormat, Model3DBounds,
5152
ListItem, CalendarEventEntry, PlanStep,
5253
WidgetRenderer, WidgetSizes,
5354
} from "./widgets.js";

‎src/render/a2ui.ts‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ import type {
2828
WidgetData, IconWidget, StackWidget, ListWidget, TableWidget,
2929
MetricWidget, MetricGridWidget, KeyValueWidget, StatusWidget,
3030
DocumentWidget, CalendarWidget, PlanWidget, EmptyWidget,
31+
Model3DWidget,
3132
} from "../widgets.js";
3233

3334
// ---------------------------------------------------------------------------
@@ -135,6 +136,7 @@ function build(w: WidgetData, ctx: BuildCtx, idOverride?: string): string {
135136
case "calendar": return buildCalendar(w, ctx, idOverride);
136137
case "plan": return buildPlan(w, ctx, idOverride);
137138
case "empty": return buildEmpty(w, ctx, idOverride);
139+
case "model_3d": return buildModel3D(w, ctx, idOverride);
138140
}
139141
}
140142

@@ -304,6 +306,32 @@ function buildEmpty(w: EmptyWidget, ctx: BuildCtx, idOverride?: string): string
304306
}, idOverride);
305307
}
306308

309+
function buildModel3D(w: Model3DWidget, ctx: BuildCtx, idOverride?: string): string {
310+
// A2UI basic catalog has no native 3D viewer. Render a Card with a poster
311+
// Image (if any) plus the format-tagged metadata. Clients that adopt a
312+
// future "Model3D" component in their catalog can branch on it; everyone
313+
// else still sees a meaningful preview.
314+
const name = w.name ?? "3D model";
315+
const inner: string[] = [];
316+
if (w.posterUri) {
317+
inner.push(emit(ctx, { component: "Image", url: w.posterUri, description: name, fit: "contain" }));
318+
}
319+
inner.push(emit(ctx, { component: "Text", text: name, variant: "h4" }));
320+
const noun = (w.format === "pdb" || w.format === "mmcif") ? "atoms" : "verts";
321+
const parts: string[] = [w.format.toUpperCase()];
322+
if (w.vertexCount != null) parts.push(`${w.vertexCount.toLocaleString()} ${noun}`);
323+
if (w.bounds) {
324+
const u = w.bounds.unit ?? "";
325+
const fmt = (n: number) => {
326+
const s = n.toFixed(n < 10 ? 2 : 1);
327+
return u ? `${s}${u}` : s;
328+
};
329+
parts.push(`${fmt(w.bounds.width)} × ${fmt(w.bounds.height)} × ${fmt(w.bounds.depth)}`);
330+
}
331+
inner.push(emit(ctx, { component: "Text", text: parts.join(" · "), variant: "caption" }));
332+
return cardWith(inner, ctx, idOverride);
333+
}
334+
307335
function formatCell(v: unknown): string {
308336
if (v == null) return "";
309337
if (typeof v === "string") return v;

‎src/render/html.ts‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ import type {
99
WidgetData, IconWidget, StackWidget, ListWidget, TableWidget,
1010
MetricWidget, MetricGridWidget, KeyValueWidget, StatusWidget,
1111
DocumentWidget, CalendarWidget, PlanWidget, EmptyWidget,
12+
Model3DWidget,
1213
} from "../widgets.js";
1314
import { escapeHtml, truncate } from "../view.js";
1415

@@ -26,6 +27,7 @@ export function renderHTML(w: WidgetData): string {
2627
case "calendar": return renderCalendar(w);
2728
case "plan": return renderPlan(w);
2829
case "empty": return renderEmpty(w);
30+
case "model_3d": return renderModel3D(w);
2931
}
3032
}
3133

@@ -215,3 +217,74 @@ function formatTime(iso: string): string {
215217
if (isNaN(d.getTime())) return iso;
216218
return d.toLocaleTimeString("en-US", { hour: "numeric", minute: "2-digit" });
217219
}
220+
221+
// ---------------------------------------------------------------------------
222+
// Model3D — 3D file viewer (glb/gltf/usdz via <model-viewer>; metadata card
223+
// fallback for formats without an embedded viewer yet).
224+
//
225+
// Loading: consumers must include the model-viewer script ONCE per page —
226+
// <script type="module" src="https://ajax.googleapis.com/ajax/libs/model-viewer/4.0.0/model-viewer.min.js"></script>
227+
// Browsers dedupe same-URL ES module imports, so multiple inclusions are safe.
228+
// ---------------------------------------------------------------------------
229+
230+
const MODEL_VIEWER_FORMATS = new Set(["glb", "gltf", "usdz"]);
231+
232+
function renderModel3D(w: Model3DWidget): string {
233+
const name = w.name ?? formatLabel(w.format);
234+
const summary = formatSummary(w);
235+
if (MODEL_VIEWER_FORMATS.has(w.format)) {
236+
const attrs = [
237+
`src="${escapeHtml(w.uri)}"`,
238+
`alt="${escapeHtml(name)}"`,
239+
`loading="lazy"`,
240+
`reveal="auto"`,
241+
"camera-controls",
242+
"touch-action=\"pan-y\"",
243+
w.autoRotate !== false ? "auto-rotate" : "",
244+
w.posterUri ? `poster="${escapeHtml(w.posterUri)}"` : "",
245+
w.cameraOrbit ? `camera-orbit="${escapeHtml(w.cameraOrbit)}"` : "",
246+
].filter(Boolean).join(" ");
247+
return `<div class="ws-model3d">
248+
<model-viewer ${attrs} style="width:100%;height:100%;min-height:240px;background:#f1f5f9;border-radius:8px"></model-viewer>
249+
<div class="ws-model3d-meta">${escapeHtml(summary)}</div>
250+
</div>`;
251+
}
252+
// Fallback: format-tagged metadata card. (PDB, URDF, STL, … will get their
253+
// own renderers later — Mol*, urdf-loader, three-stl-loader.)
254+
return `<div class="ws-model3d ws-model3d-fallback">
255+
<div class="ws-model3d-poster">
256+
${w.posterUri ? `<img src="${escapeHtml(w.posterUri)}" alt="${escapeHtml(name)}">` : `<span class="ws-model3d-format">${escapeHtml(w.format.toUpperCase())}</span>`}
257+
</div>
258+
<div class="ws-model3d-info">
259+
<div class="ws-model3d-name">${escapeHtml(name)}</div>
260+
<div class="ws-model3d-meta">${escapeHtml(summary)}</div>
261+
<a class="ws-model3d-link" href="${escapeHtml(w.uri)}" target="_blank" rel="noreferrer noopener">Open ${escapeHtml(w.format.toUpperCase())} →</a>
262+
</div>
263+
</div>`;
264+
}
265+
266+
function formatLabel(f: Model3DWidget["format"]): string {
267+
return ({
268+
glb: "glTF binary", gltf: "glTF", usd: "USD", usdz: "USDZ",
269+
stl: "STL", obj: "OBJ", ply: "PLY",
270+
urdf: "URDF", mjcf: "MuJoCo XML",
271+
pdb: "Protein Data Bank", mmcif: "mmCIF", sdf: "SDF",
272+
step: "STEP",
273+
} as Record<string, string>)[f] ?? f.toUpperCase();
274+
}
275+
276+
function formatSummary(w: Model3DWidget): string {
277+
const parts: string[] = [w.format.toUpperCase()];
278+
if (w.vertexCount != null) {
279+
parts.push(`${w.vertexCount.toLocaleString()} ${w.format === "pdb" || w.format === "mmcif" ? "atoms" : "verts"}`);
280+
}
281+
if (w.bounds) {
282+
const u = w.bounds.unit ?? "";
283+
const fmt = (n: number) => {
284+
const s = n.toFixed(n < 10 ? 2 : 1);
285+
return u ? `${s}${u}` : s;
286+
};
287+
parts.push(`${fmt(w.bounds.width)} × ${fmt(w.bounds.height)} × ${fmt(w.bounds.depth)}`);
288+
}
289+
return parts.join(" · ");
290+
}

‎src/render/markdown.ts‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ import type {
99
WidgetData, IconWidget, StackWidget, ListWidget, TableWidget,
1010
MetricWidget, MetricGridWidget, KeyValueWidget, StatusWidget,
1111
DocumentWidget, CalendarWidget, PlanWidget, EmptyWidget,
12+
Model3DWidget,
1213
} from "../widgets.js";
1314
import { truncate } from "../view.js";
1415

@@ -26,6 +27,7 @@ export function renderMarkdown(w: WidgetData): string {
2627
case "calendar": return renderCalendar(w);
2728
case "plan": return renderPlan(w);
2829
case "empty": return renderEmpty(w);
30+
case "model_3d": return renderModel3D(w);
2931
}
3032
}
3133

@@ -117,6 +119,24 @@ function renderEmpty(w: EmptyWidget): string {
117119
return w.message ?? "(empty)";
118120
}
119121

122+
function renderModel3D(w: Model3DWidget): string {
123+
const name = w.name ?? "3D model";
124+
const parts: string[] = [w.format.toUpperCase()];
125+
if (w.vertexCount != null) {
126+
const noun = (w.format === "pdb" || w.format === "mmcif") ? "atoms" : "verts";
127+
parts.push(`${w.vertexCount.toLocaleString()} ${noun}`);
128+
}
129+
if (w.bounds) {
130+
const u = w.bounds.unit ?? "";
131+
const fmt = (n: number) => {
132+
const s = n.toFixed(n < 10 ? 2 : 1);
133+
return u ? `${s}${u}` : s;
134+
};
135+
parts.push(`${fmt(w.bounds.width)} × ${fmt(w.bounds.height)} × ${fmt(w.bounds.depth)}`);
136+
}
137+
return `🧊 **${name}** — ${parts.join(" · ")}`;
138+
}
139+
120140
// helpers
121141
type PlanStep = PlanWidget["steps"][number];
122142

‎src/render/mcp_apps.ts‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,15 +82,25 @@ interface BundleArgs {
8282
}
8383

8484
function buildBundle(a: BundleArgs): string {
85+
// If the rendered HTML uses <model-viewer>, lazy-load Google's reference
86+
// implementation in the bundle. Same dedupe-by-URL behaviour browsers
87+
// give to top-level page loads — safe to include unconditionally per
88+
// bundle even if the host already loaded it.
89+
const needsModelViewer = a.innerHtml.includes("<model-viewer");
90+
const modelViewerScript = needsModelViewer
91+
? `\n<script type="module" src="https://ajax.googleapis.com/ajax/libs/model-viewer/4.0.0/model-viewer.min.js"></script>`
92+
: "";
93+
8594
return `<!DOCTYPE html>
8695
<html lang="en">
8796
<head>
8897
<meta charset="utf-8">
89-
<title>${escapeHtml(a.title)}</title>
98+
<title>${escapeHtml(a.title)}</title>${modelViewerScript}
9099
<style>
91100
:root { color-scheme: light dark; }
92101
body { margin: 0; font: 14px/1.5 ui-sans-serif, -apple-system, "Helvetica Neue", Arial, sans-serif; background: transparent; }
93102
#scenecast-root { padding: 16px; }
103+
model-viewer { width: 100%; min-height: 240px; background: #f1f5f9; border-radius: 8px; }
94104
${a.extraCss}
95105
</style>
96106
</head>

‎src/types/index.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,12 +12,14 @@ export { Contact, type ContactRecord, type ContactsState } from ".
1212
export { Event, type CalendarEventRecord, type CalendarEventsState } from "./event.js";
1313
export { Task, type TaskRecord, type TasksState } from "./task.js";
1414
export { Document, type DocumentRecord, type DocumentState } from "./document.js";
15+
export { Mesh, type MeshFileState } from "./mesh.js";
1516

1617
import { Email } from "./email.js";
1718
import { Message } from "./message.js";
1819
import { Contact } from "./contact.js";
1920
import { Event } from "./event.js";
2021
import { Task } from "./task.js";
2122
import { Document } from "./document.js";
23+
import { Mesh } from "./mesh.js";
2224

23-
export const canonicalTypes = { Email, Message, Contact, Event, Task, Document } as const;
25+
export const canonicalTypes = { Email, Message, Contact, Event, Task, Document, Mesh } as const;

0 commit comments

Comments
 (0)