Inline image rendering via kitty graphics protocol (generic placeImage API)#2325
Inline image rendering via kitty graphics protocol (generic placeImage API)#2325Agrejus wants to merge 14 commits into
Conversation
This introduces inline image support in compose mode, leveraging the Kitty graphics protocol for terminals that support it (Kitty, WezTerm, Ghostty, Konsole). Plugins can now place raster images (PNG, JPG, etc.) and dynamically rendered diagrams (Mermaid via `mmdc`) directly into buffer content, anchored to text positions. The editor manages image transmission, cleanup, and renders placeholder cells that scroll with text. The `markdown_compose` plugin uses this to render image links and Mermaid diagrams. A new `editor.getGraphicsCapability()` API allows plugins to detect terminal support and skip expensive rendering work when images won't be displayed. Additionally, compose mode now provides lightweight, language-agnostic syntax highlighting and background tinting for fenced code blocks.
…xternal plugin The branch now exposes only generic, content-agnostic core APIs (placeImage/clearImages, getGraphicsCapability, readFileBytes, the kitty Unicode-placeholder render path). Markdown-specific rendering (inline images, mermaid, fenced-code highlighting) lives in the standalone fresh-markdown-preview plugin, built entirely on those APIs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…code wells - Cache fenced-code-block byte ranges per buffer and skip all markdown processing (tables, emphasis, conceals, table soft-breaks) for lines inside them. Fixes |-leading code lines (e.g. TS union types) growing table borders inside ``` fences. - Conceal ATX heading markers (revealed on cursor line) and style heading text by level (bold/underline/color — terminals can't change font size). - Conceal fence marker lines (```lang / closing ```) when the cursor is elsewhere, so code blocks render as a clean well. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Raw kitty graphics APC sequences are silently stripped by tmux, so inline images never reached the outer terminal. When $TMUX is set, wrap each transmit/delete in tmux DCS passthrough (ESC P tmux ; … ESC \, doubling inner ESCs). Requires 'set -g allow-passthrough on'. Unicode-placeholder cells already ride through as normal text, so only the data transmit needs wrapping. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
VirtualTextManager stores entries in a HashMap, and the range queries sorted only by (position, priority) — entries tying on both came back in arbitrary iteration order. For placed inline images, whose N reserved rows all share one anchor and priority 0, the rows rendered shuffled: diagrams displayed as torn horizontal bands. Tiebreak by VirtualTextId (a monotonic counter, so id order is insertion order) in query_range, query_lines_in_range, and query_inline_in_range. This fixes image row order without reserving the priority space, and makes every plugin's equal-priority virtual texts deterministic. Also commit .gitignore for macOS .DS_Store files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
markdown_compose rebuilds each line's conceals by clearing the line's byte range first — but clearConcealsInRange drops conceals from EVERY namespace, so it silently destroyed other plugins' conceals on those lines (e.g. an external preview plugin collapsing rendered diagram source). Same failure class as the unscoped overlay clear fixed in issue sinelaw#2146; conceals never got the namespaced variant. Add ConcealManager::remove_in_range_for_namespace and plumb it through as clearConcealsInRangeForNamespace (PluginCommand, dispatch, QuickJS binding, fresh.d.ts, ts export list). markdown_compose now clears only its own md-syntax conceals per line. Also document two non-obvious choices in markdown_compose: the double refreshLines on enable (instant paint + corrected repaint once the fence cache exists) and the uncapped whole-buffer read in rebuildFenceRanges (fences are cross-line state; stale ranges degrade gracefully if a cap is ever added). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
new_from_env existed only so tests could construct a manager with passthrough deterministically off — a confusing split for one call site. Detection ($TMUX set => wrap escapes) now lives in new() itself, and the byte-exact tests pin set_tmux_passthrough(false) explicitly, which states their assumption instead of relying on which constructor was called. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… rules Block quotes render as a dimmed vertical bar (one width-preserving bar glyph per '>', nesting included), list bullets as '•', task boxes as ☐ / ☑, and horizontal rules as a '─' line spanning the compose width. All markers reveal while the cursor is on the line, matching the existing heading/fence behavior, and fall through to inline emphasis processing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
In-text references ([^1]) conceal to Unicode superscript digits (¹) in link color; non-numeric labels keep a compact ^label form. Definition lines ([^1]: text) collapse the marker to the same superscript and dim the definition text, mirroring GitHub's footnotes section. Both reveal while the cursor is on the line. Going through findInlineSpans means table widths and soft-wrap budgets account for the concealed syntax automatically. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Table column widths, cell padding, truncation, and wrapping all measured cell content with String.length (UTF-16 code units), so wide glyphs threw off alignment: an emoji like ✅ is one code unit but two terminal cells, which over-padded every cell containing one and pushed the right border past the table outline. Add a wcwidth-style displayWidth covering East Asian wide ranges, the emoji planes, emoji-presentation symbols (U+2300–U+2BFF, where ✅/❓ live), zero-width joiners/combining marks, and the U+FE0F variation selector, and route all width math through it. Also conceal alignment colons in separator rows so ':---:' no longer bleeds through the rendered rule. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wide tables previously either truncated each cell to one line or split the single source line with soft breaks at whatever spaces happened to line up — which couldn't keep independent columns aligned, bled cell text across row boundaries, and corrupted neighbouring lines. Render tables the way GitHub does instead: each column word-wraps to its width and the row spans as many visual lines as its tallest cell. Mechanism: a source row's first visual line is rendered in place (cells concealed to their first wrapped fragment, pipes -> box-drawing); the overflow lines are emitted as virtual continuation lines below the row, the same primitive the borders already use. Every source row stays one source line, so cursor mapping and editing are unaffected and alignment is computed from generated text rather than fighting the byte layout. Also in table layout: - Distribute column widths by water-filling (max-content with fair shrinking) instead of proportionally. The old split starved short label columns to feed an already-wide prose column, chopping single words like `workflowTransitions` mid-word. Now narrow columns keep their natural width and only the genuinely wide prose column shrinks. - Cap the frame at the configured compose width (or `maxWidth` when unset) and leave a one-cell right margin, so a table on a wide terminal reads like a README instead of stretching edge-to-edge, and its right border never lands in the final column (which auto-wraps). Two correctness fixes surfaced by real documents: - Never let one line abort the whole batch. The per-line debug log sliced lineContent at 40 UTF-16 units, which can cut an astral char (e.g. the 🟡 status emoji at U+1F7E1) between its surrogate halves; the resulting lone surrogate failed the host's string->UTF-8 conversion and threw, aborting the lines_changed loop so that line AND every line after it rendered as raw markdown. Slice by code points, and wrap each line's processing in try/catch so a single bad line can no longer cascade. - Count colored circles/squares (U+1F7E0..U+1F7EB, e.g. 🟡) and regional indicators as two cells in displayWidth — they sat in a gap between the existing emoji ranges, so a 🟡 was measured as one cell but drawn as two, shifting every border to its right by one. - Stop revealing a line's concealed markers when the cursor sits at the start of the *next* line: byteEnd points just past the trailing newline, so a cursor there counted as "on" the line and left a heading's `##` visible while editing the blank line below it. Adds e2e tests: clean wide-table frame, full composition of a table taller than the viewport after scrolling, and a wrapping row with an astral emoji not aborting composition. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Format the table-rendering e2e tests added in the previous commit: rustfmt breaks the long `harness.send_key(...).unwrap()` chains and over-width string/constructor lines that CI's `cargo fmt --all --check` flagged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rders Two compose-mode markdown rendering fixes: - Escaped pipes (`\|`) inside table cells were split as column separators, fanning a row into phantom columns and skewing column widths. Split table rows on unescaped pipes only and render `\|` as a literal `|`. Adds escape-aware helpers (isEscapedPipe / tablePipePositions / tableRowInner / splitTableCells) used by every table cell-split and width path. - Stray table-border `│` glyphs were left hanging on blank lines when scrolling inside tmux. The per-frame synchronized-update markers (DEC mode 2026) were sent to the outer terminal unwrapped; tmux's handling drops individual cell-clear updates from a frame diff, leaving stale glyphs until a full redraw. Skip the markers when inside tmux (TMUX set) — tmux batches its own pane refreshes, so they buy nothing there. Tests: escaped-pipe table regression test, plus a scroll probe that guards the composed buffer against logical stray-border regressions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
This is a big PR doing several different things. I need to think about the graphics and kitty protocol stuff and allowing plugins directly to read files, but the PR does a few other things that are easier to merge. Can we break into a separate PR doing only the "remove_in_range_for_namespace" and the changes in virtual_text.rs that are NOT related to the graphics stuff? It will be easier to know we aren't breaking anything? i.e. anything not related to graphics - if possible to break that into a separate PR we can merge first |
| * *cells*, not code units — an emoji like ✅ is one UTF-16 unit but occupies | ||
| * two cells, so `.length`-based padding pushes the right border out of line. | ||
| */ | ||
| function charDisplayWidth(cp: number): number { |
There was a problem hiding this comment.
This should be a replaced by a core function exposed to plugins, and since we already do unicode width in the rust core, the implementation should use the same underlying unicode logic we have in the rust core instead of duplicating it
Yes, good call. I'll start breaking it down now and close this one |
|
Closing this in favor of smaller, focused PRs, per the review above:
Breaking it up so the easy, low-risk, non-graphics pieces can land first and the graphics/kitty + plugin-file-read work can get its own dedicated review. First split is up now: #2399 — Namespaced conceal range-clear + deterministic virtual-line ordering. That's exactly the Planned follow-ups (each its own PR):
Closing now; please direct review to #2399. |
|
@sinelaw — the non-graphics work from this PR is now split into three small, independent PRs, each branched from
Still held back for your separate review (not yet opened): the graphics / kitty inline-image rendering and the plugin file-read APIs. The markdown-compose rendering will follow as its own PR once #2399 and #2401 land (it depends on both, and is where |
…h/stringWidth)
Adds a single source of truth for terminal display width in `fresh-core`
(`display_width::{char_width, str_width}`, backed by the `unicode-width`
crate) and exposes it to plugins as `editor.charWidth(codePoint)` and
`editor.stringWidth(text)`.
Why: plugins currently have to re-derive their own Unicode width tables (e.g.
markdown_compose's hand-rolled `charDisplayWidth`), which inevitably drift from
how the editor itself measures cells. With this, a plugin measures width with
the exact same logic the renderer uses for layout and cursor positioning — so
column widths, wrapping, and alignment agree by construction.
- `fresh-core::display_width` is now canonical; `fresh-editor`'s
`primitives::display_width` re-exports `char_width`/`str_width` from it
(its byte/column helpers and the `DisplayWidth` trait are unchanged), so
there's one implementation, not two.
- The plugin runtime calls the core function directly (pure, no command
round-trip), matching the existing synchronous backend methods.
No behavior change to the editor; this only adds APIs. The follow-up
markdown-compose PR will use these to delete its duplicated width table
(addresses @sinelaw's review note on #2325).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2 participants
Motivation
I wanted to open a README in Fresh and see it rendered roughly the way GitHub or Cursor would show it, instead of staring at raw markdown. Opening it in an external viewer works, but Fresh already has the rendering machinery to do this in place — so a command-palette action that previews markdown without ever leaving Fresh felt like a natural fit.
Rather than baking markdown support into the core, this PR exposes the underlying capability as a generic API: any plugin can place rendered images, whether that's markdown, diagrams, plots, or anything else it can rasterize. The markdown/mermaid preview is just the first consumer.
Summary
add
placeImage/clearImages/getGraphicsCapabilityplugin APIs: plugins place raster images (PNG) anchored to buffer byte positions; the core reserves virtual placeholder rows and renders kitty Unicode placeholders over them, so images scroll with text and survive the cell-diff rendererdetect graphics capability from the environment (kitty / WezTerm / Ghostty / Konsole;
FRESH_GRAPHICSoverride), shared between core and plugin runtime so both always agreewrap transmit/delete escapes in DCS passthrough inside tmux (requires
allow-passthrough on); placeholder cells ride through as normal textfix non-deterministic virtual-line ordering: equal-priority range-query results rendered in arbitrary order (an image's N reserved rows came back shuffled, displaying diagrams as torn bands); queries now tiebreak by insertion order
add
clearConcealsInRangeForNamespaceso one plugin's per-line rebuild no longer destroys other plugins' conceals (same motivation as the namespaced overlay clear from Markdown compose (preview) mode hides LSP diagnostic higlighting, gutters #2146); markdown_compose switched to itmake markdown_compose fence-aware: cache fenced-code-block ranges per buffer and skip markdown processing inside them (fixes
|-leading code lines growing table borders); conceal fence/heading markers with cursor-line revealrender block quotes (
>→ dimmed▌bar, nesting included), list bullets (•), task checkboxes (☐ / ☑), horizontal rules (full-width─), and footnotes ([^1]→ superscript ¹, with dimmed definition lines) in markdown_compose — these constructs were parsed for indentation but displayed as raw sourcerender markdown tables as aligned, word-wrapped frames in markdown_compose: distribute column widths by water-filling (narrow label columns keep their natural width; only wide prose columns shrink), wrap long cells into virtual continuation rows so a row spans as many visual lines as its tallest cell — like a rendered README table — instead of truncating or splitting the source line, cap the frame at the compose width with a one-cell right margin so it neither stretches edge-to-edge nor overflows the final column, and count colored-circle/square status emoji (U+1F7E0..U+1F7EB, e.g. 🟡) and regional indicators as two cells so borders stay aligned; also harden the per-line compose loop (code-point slicing + try/catch) so one malformed line — e.g. an emoji sliced between its surrogate halves — can no longer throw and leave that line and every line after it rendered as raw markdown
Compatibility
Test plan
cargo fmt --all -- --checkcargo clippy --all-features --all-targetscargo nextest run --no-fail-fast --locked --all-features --all-targets(CI: ubuntu / macos / windows)cargo test --test e2e_tests markdown_compose(67 passed, incl. table wrapping/alignment, tall-table scroll, and emoji-row composition regression tests)Demo
https://www.loom.com/share/af49ee3975024c2884346282859fc9c3
Exercised end-to-end in Ghostty + tmux 3.6a with an external markdown-preview plugin: inline images, mermaid diagrams, and a fullscreen zoom view.
🤖 Generated with Claude Code