Infinite canvas / engine architecture questions

[christoph-busse]

TL;DR: I’m building a browser-first infinite canvas/editor engine (Rust + WASM + WebGPU/Vello) with a worker-based architecture and deterministic ops/log document model (collab + offline-first). I’m optimizing less for raw vector draw speed and more for indexing/invalidation + smooth large edits (duplicate/transform thousands of nodes) as scenes scale. I’m posting 6 architecture questions below; any pointers welcome. Also: if you know other real-time collaborative raster editors (besides magma), I’d love links.

Hi @xiaoiver, thanks for offering to reply here.

Hey everyone, I am Christoph (Germany). I’m building a browser-first infinite canvas / editor engine:

• Rust + WASM + WebGPU (currently Vello for vector rendering)
• Main thread: React/DOM shell (input capture, IME textarea, clipboard, file dialogs)
• Engine runs in a Worker: OffscreenCanvas + WebGPU device/queue, scheduling, hit-testing/indexing, derived caches
• Deterministic pipeline + ops/log-based document model (collab-oriented and offline first workflows (linear/figma like))

I would appreciate any thoughts on the following topics, can be off-topic too. Doing this more as a hobby at the moment and I still need to learn a lot.

Below I’ll post six architecture questions as separate comments (one per topic) so replies can stay threaded and searchable, never really used github discussions before, but I hope it helps.

Why I’m asking / what I’m optimizing for

I’m less worried about “raw rasterisation speed” (for vector work Vello is already fast enough), and more about:

• spatial indexing + invalidation as scenes grow,
• “what needs rebuilding?” logic without over-invalidation,
• smooth interaction under heavy edits (duplicate/move/transform many nodes),
• a foundation that can later support raster, images, compositing, etc.

Very rough editor stress tests I ran (anecdotal, not a scientific benchmark)

I’ve been testing a lot of modern editors to calibrate what “good” feels like. My basic stress tests are usually:

• create many simple vector primitives (mostly rectangles/circles),
• duplicate a large selection,
• transform/move large selections,
• disable extra expensive features if possible (snapping, guides, geometry constraints, etc.) because they can dominate results and make comparisons misleading.

A few anecdotal observations from my machine Macbook Pro M4 (again: rough, likely influenced by UI/DOM/layout, data models, snapping, etc.):

• Figma starts to feel slow around ~5k+ vector nodes for heavy operations (e.g. large duplication / transforms), preview transform keeps stable for single nodes (some kind of preview / proxy system is likely in place, or index chunking). Note: Figma is still a crazy piece of software, it is one of the best web engines there is. But from reading the forum and my own experience I can still see how performance could be better in certain workflows (big edits, libraries, huge node counts, complex prototypes can tank Figma at scale)
• In Figma duplicating tens of thousands of shapes can take very long (a minute +, probably db overhead, repack time) (e.g. duplicating ~40k → ~80k).
• Illustrator, Inkscape are more performant in these ops, looks like some kind of network db overhead maybe in Figma these local tools do not have. Or just better APIs and native optimizations.
• Illustrator Web felt surprisingly fast in my quick tests (seemed to use workers/multithreading effectively).
• Graphite a open-source editors I tested degrades very early (sometimes in the low hundreds of shapes), likely due to main-thread work or the missing worker/offscreen architecture freezing the entire UI. Could also be the missing invalidation setup or the complex node foundation from what I saw in the code. Penpot has/had similar issues, but the new Skia based architecture made improvements on the html dom canvas that was used before.
• I tested more obscure tools like absl.design I found on Reddit (Zig / Skia) or grida.co, always happy to see some others working on this.

I am still looking into performing a better cross-tool benchmark that evaluates different ops more deterministic. Adding Sketch, Affinity, Linearity and more tools for a more complete picture, also adding Vector tests in Clip Studio, Photoshop or other Raster tools with vector capabilities.

• A few tools are hard to test fairly because snapping/constraints (snap to geometry, to object) couldn’t be disabled and dominated cost even at low shape counts. (I think most of them can do better in view based constraints on these calculations since they perform bad even at a zoomed in state, where frustum culling and other optimizations should reduce overhead, but maybe a flat index is still the issue, since most of these tools struggle with high flat node counts. (Flat as in no groups, frames, so just nodes at root. I think groups really help in isolating and handling more shapes)

I’m sharing this not to “rank” tools, but to give context for the kinds of bottlenecks I’m trying to avoid (especially duplication/transform + maintaining correct z-order and hit-test behavior under large edits.)

Longer-term target

My near-term “stretch goal” is handling ~1,000,000 unique primitive vector shapes on an infinite canvas with reasonable interaction performance (I agree real-world documents often have fewer nodes, but I want a strong base, since I am long time user of most of these tools and I can feel the bad performance issues... Nothing is slowing down the creative process than bad fps). Very long term I am looking into unifying all kinds of editors into a realtime collaborative cross platform editing suite, like all adobe apps in one. I think it is possible ¯(°_o)/¯

Extra ask if anyone reads this long text (community)

If anyone knows other real-time collaborative raster editors (besides magma.com), I’d love links/pointers. I’m trying to study viable architectures for “raster + collaboration” early. But from what I found these are none existent. Same goes for CRDT realtime Video editors (has similar decode/encode complexity). Or even audio platforms (I saw a few), but essentially a DAW in Web with realtime editing.

I’ll post a few questions below as separate comments. Feel free to answer only what you have time for, partial pointers are very welcome. I am appreciating any kind of inputs honestly.

[christoph-busse]

1) Scenelets/chunking vs a strong spatial index (and delaying chunked artifacts)

Vello is already fast for me (e.g. ~90k circles at 60+ FPS). My bigger worry is indexing + invalidation:

• hit-test + visibility queries,
• “what needs rebuilding?” under edits,
• keeping interactions smooth when scenes get large.

I started integrating a scenelet/artifact/chunking system (I am already deep into it at this point), but it’s hard and adds overhead/complexity. One idea is:

• keep chunked artifacts thin initially (or even delay them),
• push a dedicated spatial index + invalidation strategy as far as possible,
• only go “deep” into chunked artifacts later once raster/images/compositing force it.

Question:
If you were building an infinite canvas today, would you commit early to a scenelet/chunk artifact architecture, or would you first maximize a simpler index-based approach and introduce chunking later? (Could be unavoidable, not sure, since I have never build anything like this I would like to hear your thoughts.)

Also: do you run stress tests/benchmarks/flamegraphs for edit ops (duplicate/transform large selections)?

[xiaoiver]

I think you'll eventually need a spatial index like dynamic R-tree or BVH. Because hit-test(If you're using color-based picking, you might not need it.) and viewport culling are important if you want to build a high-perf infinite canvas.

Scene graph
        ↓
Spatial index
        ↓
Culling result
        ↓
Vello Scene builder
        ↓
Render

[xiaoiver]

"invalidation strategy" is also needed I think, kind of "dirty flag"?

visible = spatial_index.query(viewport);
for node in visible {
    if node.dirty_any || parent_dirty_chain {
        build(node);
    }
}

[xiaoiver]

I notice Graphite built on vello also use viewport culling like this:

// simplified from portfolio_message_handler.rs
fn render_document(&mut self, document_id: DocumentId) {
    let doc = self.documents.get(&document_id).unwrap();
    
    let viewport_rect = doc.navigation_handler.visible_rect();
    let scene = doc.executor.evaluate_graph(
        document_id,
        &doc.network_interface,
        viewport_rect,
    );
   
    let mut vello_scene = Scene::new();
    for layer in scene.visible_layers {
        layer.render_to_vello(&mut vello_scene);
    }
    
    self.render_vello_scene(vello_scene, &doc.viewport);
}

[christoph-busse]

I am currently doing both. R-tree (rstar) for hit-testing and visibility queries, plus a coarse-grid scenelet system (1024 doc-unit cells, not sure if this size stays will maybe adjust if tiling work shows unification possiblities) that maps nodes → grid cells. The scenelet layer gives me O(changed + visible chunks) rebuilds, and the R-tree handles broad-phase culling/picking. So far the combo works well, the scenelet overhead is "manageable" and it gives a clean path toward per-chunk artifacts later (raster, images, compositing). I'd say: if you're building for scale, having the spatial index alone is great, but scenelets pay off once you start thinking about incremental rebuilds and cache invalidation. But honestly it is quite complex settingup policy, evection and invalidation, running in some weird bugs...

[christoph-busse]

2) Text pipeline design: one canonical pipeline vs multiple specialized ones

I want to add text soon (shaping + layout). I read your text rendering lesson (shaping/layout + SDF/MSDF atlas).

I’m trying to avoid building two permanent text systems.

Question:
Would you recommend one canonical shaping/layout pipeline that produces a single output format, and then swapping the rendering backend (SDF/MSDF + emoji, etc.)?

Or is it worth maintaining a separate specialized pipeline in practice? Where do you draw the boundary (layout/shaping vs rasterization vs atlas vs rendering)?

[xiaoiver]

I highly recommend using a wrapped up engine, such as cosmic-text which use:

• Font shaping (using HarfRust) including RTL and bidirectional rendering
• Font layout
• Font rendering swash, including Color emoji

COSMIC Text provides advanced text shaping, layout, and rendering wrapped up into a simple abstraction.

When I tried to implement these features on the web, I found there was no out-of-the-box Web API, not even for measuring text size. My current implementation has numerous bugs.

[xiaoiver]

I found swash does not compute a SDF by default. So maybe we still have to use EDT to generate SDF from a grayscale bitmap. If you want to build a zoomable canvas, I think SDF / MSDF is still necessary.

[christoph-busse]

Today I am finally starting text scafolding work (planning this took ages...). I have looked at a lot now and I think the following is the best solution for me: A dedicated text crate built on Parley (layout) + Skrifa/Fontations (fonts/outlines) + HarfRust (shaping) + Swash (glyph raster, rather tactical usage, I think adoptiong it to broadly would not work for everything in my case). Cosmic-text is solid like you suggested, but I need more control over layout modes (code lines, canvas box, threaded frames, paged flow, path text. I have a lot of features planned needing complex baseline. So I went with a lower-level stack. Still one canonical shaping/layout pipeline though, just swapping rendering output. This is also more in line with where vello / google are headed. Its a good sing for me if Google Chrome adopts a similar stack and that support will be long term.

[christoph-busse]

3) Raster brushes + collaboration: ops-only canonical state + deterministic local reconstruction

For raster painting, I’m considering keeping canonical state as ops/intent only:

• store brush stroke ops + parameters in the document model,
• each client deterministically reconstructs derived raster caches locally,
• later add derived checkpoints for fast join/recovery if needed.

I’m tempted to go GPU-first for the brush core (compute/render pipelines), but I know cross-device bit-exact determinism on GPUs is hard. Magma, Krita and others all do CPU brushes from what I can see. I think Procreate Silica engine runs GPU accelerated ones (but they clip to max canvas size (easier than infinite canvas), probably a Apple Silicon Metal limitation, not sure I haven't looked into it too deep yet).

From what I can infer, Magma pushes image work into a WASM worker, uses a small-grid dirty-rect approach 100x100, and relies on specialized compression formats (QOIR (QOI-Rects for fast lossless) and GFWX (Graph/Good Fast Wavelet codec for lossy)) for fast transfer/storage. They also use SMID for CPU acceleration. Performance wise I found it really good overall. Its pretty impressive.

Question:
Does “ops-only canonical + local reconstruction” sound sane long-term for collaborative raster? I think magma merges raster data and dosen't do local reconstruction (but not really sure).

And would you go GPU-first for the brush core, or CPU-first with selective acceleration? Any practical advice for keeping this reliable on the web?

[christoph-busse]

4) Many images: pyramids / virtual-texture direction + non-destructive edits

Long-term I want to support lots of images on the canvas:

• thumbnails + mip pyramids,
• possibly “virtual texture”-style streaming (load only what’s needed for the current view).

I briefly looked at KTX2 as a GPU container, but:

• it’s expensive to encode,
• and I’m unsure how it fits with Lightroom-like non-destructive edits (levels/curves/saturation) vs destructive raster workflows (paint/warp). I think non-destructive edits might be fine, but for destructive raster workflows it might not fit well.

One idea:

• keep originals as source-of-truth (dedupe hash...),
• store edit parameters,
• apply adjustments shader-side in realtime,
• generate pyramids/compressed variants incrementally in background (or for export).

Questions:
Is virtual-texture-style image pyramids on WebGPU realistic in your view, have you seen anything resembling this?

(I found Google Earth to be a good example for what I want to achieve here (without the sphere 🌏). BUT it is static generated and then streamed, not real time editable)

And are compressed GPU containers like KTX2 still useful if users expect editable adjustments?

[christoph-busse]

5) Implementation choice: JS vs Rust/WASM (and messaging formats)

I noticed you’re building a lot in JavaScript and I’m curious what trade-offs mattered in practice.

Questions:

• What made you choose JS over Rust/WASM for core engine parts (iteration speed, debugging, ecosystem, portability, etc.)?
• For main-thread ↔ worker communication: did you prefer JSON-like messages for ergonomics, or binary messages for throughput/GC?

My intuition: latency isn’t the limiting factor for most interactions (outside brushes / high-frequency streams), but I’d love to hear what actually mattered in real usage.

[xiaoiver]

I feel I'm stuck in a path dependency—I'm more familiar with browser-side technologies, while my Rust-related skills remain relatively weak. But I'm very interested with vello.
However, I believe rendering can be entirely handled within the Worker (as demonstrated in my simple experiment using OffscreenCanvas), while the main thread manages interaction (such as listening for user mouse events) and DOM-related tasks (rendering HTML/embedded content). Communication latency between the two is negligible.

[christoph-busse]

Agree, rendering in the Worker with OffscreenCanvas works great and latency is negligible. I think it works best when most work stays inside the worker island. Frequent JS ↔ WASM boundary crossings can eat your gains fast, though SharedArrayBuffer helps.

Relevant recent post: They rewrote their Rust/WASM parser in TS and got 3x faster, in this case their Rust parsing was fast but the boundary (string copies, serialization) dominated cost. WASM work best when we hand it bytes, let it do heavy work (decode/encode, geometry, scene builds), and return a compact result. Perfect for canvas engine

[christoph-busse]

6) AI agent integration: overlay feature vs foundational engine decision

I saw you started integrating an AI agent. I want an agent to be a core tool too (commands, editing assistance, automation), ideally driven by the same operation protocol the editor uses.

Question:
In your experience, is the agent mostly an “overlay feature” once the ops/command protocol is solid?

Or are there engine/design decisions you wish you had made earlier to make agent integration clean (e.g., richer semantic ops, better introspection, deterministic replay hooks, better debug surfaces)?

And maybe you have some thoughts on diff / review management in an infinite canvas system. I am currently working this out, one should always be able to see what has changed in Canvas/Nodes/Texts or other related editor parts, I am not aware of any kind of tool that does this. Maybe something like a simple before / after preview or a diff shape node that looks different and exits at the old and new position at the same time.

[xiaoiver]

I'm more focused on AI-generated/edited images in that section.
But I've also been studying the pencil.dev data structure lately, which is highly AI-friendly. In the demo, we can see Agent teams manipulate all the shapes in canvas.

[xiaoiver]

As for CRDT diff, both loro.dev and Yjs can handle these Map/Array structures quite well.

[christoph-busse]

I had my eye on pencil.dev also, haven't had time yet to check it out. I think the new Google Stitch falls into the same AI agent manipulating the structure kind of thing.

I think if the ops protocol is solid, the agent is mostly an overlay, just another "user" or the same as the real "user" emitting the same ops. The harder part is making ops expressive enough (semantic intents over granular mutations) and giving the agent a good query/introspection surface to "see" canvas state. I see a possible way to train a diffusion model on user inputs that way the agent could get better over time with a fine-tuned model. I think at some point a fine-tuned model makes sense even if it is not custom diffusion, maybe fine-tuned Qwen or other open source that has a better grasp over the engine API, this way it might be more efficient. I think it could get quite token-hungry if it always needs to load the entire "how does this interface work" stuff into context, and also slower. Cursor did the same recently with the Composer 2 Kimi 2.5 fine-tune.

On canvas diffs. haven't seen any tool do this well yet. I'm thinking about treating the ops log like git treats commits but spatially: before/after split view, ghost nodes at old positions, or a timeline scrubber over the log. The CRDT history gives you the data, the hard part is making it visually useful on a canvas. Not sure how feasible this is, but I'll try.

And on the CRDT thing, I did choose loro.dev since it might handle larger text blobs a bit better. Seeing that Figma adopted a similar inspired algo for their own CRDT sync I don't think its a bad choice, but I feel like Yjs has larger adoption.

[xiaoiver]

I'm trying to use vello as rendering engine and looks good now!
https://infinitecanvas.cc/guide/lesson-035

[christoph-busse]

Excited to see you picking up Vello!