Merge pull request #2059 from virtualcell/feat/ui-macro-recorder

Record UI interactions as replayable scripts, for demos and help pages

Author: jcschaff

.claude/skills/swing-debug/SKILL.md

@@ -79,6 +79,37 @@ pre-redirect lines. `bridge.sh log 200` serves the tail.
   fire menu items from the models, opening no popup at all.
 - `JTree` rows report the text the **renderer** draws, so VCell's database trees
   read as "Lee 2026 Systems-level…" rather than `PublicationInfo@415f5f4f`.
+- **Not everything in a menu bar is a menu.** VCell puts icon-only controls straight into
+  it — the detach toggle is a `JMenuItem` whose entire label is a tooltip — so `/menu`,
+  which matches items by visible text, cannot address them. Click those by `name=`.
+
+## Recording a flow instead of writing one
+
+`bridge.sh record start` … drive the UI … `bridge.sh record stop <file.json>` captures what
+you did as an editable script, replayed with `bridge.sh replay <file.json>`. Good for
+tutorials and demos, and for turning a manual repro into a scenario.
+
+Three things decide whether it works, and all three have bitten:
+
+- **Both routes are recorded**: real input via the AWT listener, and the bridge's own
+  model-based endpoints (which post no event) by reporting themselves. Robot helpers do not
+  double-report. The flip side is that **scripted setup done while recording lands in the
+  script** — do it before `record start`, or pass `--no-bridge-actions`.
+- **A modal dialog blocks input to everything behind it**, so clicks aimed at the main
+  window produce no events at all. The version-mismatch warning on a source build appears a
+  moment *after* the menus do — `wait` for it, don't look once. `record status` distinguishes
+  the two failures: `rawEvents: 0` means nothing was seen, not that nothing was captured.
+- **Replay is patient where a scenario has to be.** It retries a step until it takes, so the
+  menu item that sits disabled for a moment after a modal dialog closes needs no retry loop
+  of your own.
+
+Keep assertions in the scenario, not the recording — replay `--from N --to N` a step at a
+time so state is checked *between* clicks. `scenarios/detach-window-recorded.sh` is the
+worked example, and passes the same 15 checks as the hand-written original.
+
+For a filmed replay use `--driver robot`: the cursor glides to each target and clicks for
+real. The default `semantic` driver moves no cursor, which is right for CI and wrong for a
+video. Same recording either way.
 
 ## Reusable recipes
 

tools/debug-bridge/README.md

@@ -7,7 +7,10 @@ so getting a live, drivable client takes one command instead of a rediscovered r
 |---|---|
 | `launch-client.sh` | Launch the client **from `target/classes`** (no packaging) with the bridge on, using the bundled install4j JRE and a cached dependency classpath. Backgrounds itself and waits for `/health`. |
 | `bridge.sh` | CLI over the bridge's HTTP endpoints: `tree`, `find`, `menu`, `wait`, `assert`, `shot`, `log`, … URL-encodes arguments, pretty-prints via `jq` when available; `wait`/`assert` exit non-zero on failure so scenarios are plain shell. |
-| `scenarios/` | Reusable end-to-end scripts built on `bridge.sh` — `smoke.sh` checks bridge, main frame, menus, EDT and screenshots using only semantic selectors. |
+| `replay.py` | Replay a recorded script (`bridge.sh replay`). Issues each step through the endpoint that already implements that verb, retries a step until it takes, and waits for a window a step opened rather than sleeping. `--from`/`--to` play a slice; `--speed`/`--max-delay` re-time it. |
+| `scenarios/` | Reusable end-to-end scripts built on `bridge.sh` — `smoke.sh` checks bridge, main frame, menus, EDT and screenshots using only semantic selectors; `detach-window.sh` and its recorded twin `detach-window-recorded.sh` drive the modeless-child-window round trip. |
+| `doc-scaffold.py` | Turn a recording plus its captures into a VCell help-page skeleton (`<vcelldoc>` XML for `UserDocumentation/originalXML`). Writes the mechanical parts — ordered steps, image references — and leaves the prose to a person. |
+| `scenarios/recordings/` | Captured UI scripts. JSON, hand-editable, replayed by `replay.py`. |
 
 ## Quick start
 
@@ -24,3 +27,73 @@ tools/debug-bridge/scenarios/smoke.sh          # end-to-end sanity
 local install at `~/Applications/VCell_Alpha` for the JRE/native libs; override with
 `VCELL_API_HOST` / `VCELL_INSTALL_DIR`. The client's real log is
 `~/.vcell/logs/vcellrun_<site>.log` — or just `bridge.sh log 100`.
+
+## Recording and replaying
+
+```bash
+tools/debug-bridge/bridge.sh record start                  # or: record start <file>
+# ... drive the UI by hand: click, type, pick menu items ...
+tools/debug-bridge/bridge.sh record stop scenarios/recordings/my-flow.json
+
+tools/debug-bridge/bridge.sh replay scenarios/recordings/my-flow.json --dry-run
+tools/debug-bridge/bridge.sh replay scenarios/recordings/my-flow.json
+DRIVER=robot ...                       # or --driver robot: the cursor visibly moves
+```
+
+**Both routes are recorded.** Real mouse and keyboard input reaches the AWT event queue and
+is captured by the listener. The bridge's own model-based endpoints (`click` on a button,
+`menu`, `settext`, `tab`, `row`, `trow`, `expand`) post no event at all, so they report the
+step themselves — a session scripted entirely with `bridge.sh` records correctly. Robot-driven
+helpers (`rbclick`, `drow`, `dtrow`, …) do not report themselves, because the listener already
+sees their real events.
+
+Two consequences worth knowing: **scripted setup performed while recording ends up in the
+script** — start with `--no-bridge-actions`, or do setup before `record start`. And if a
+recording still comes out empty, `bridge.sh record status` reports `rawEvents`, which
+separates "saw nothing" (a modal dialog blocking input will do that) from "captured nothing".
+
+**The file is written as you go.** `record start` creates it immediately and rewrites it
+after every step, so a client that crashes — or that you kill — costs at most the step in
+progress, not the take. `record start <file>` chooses where that happens; `record stop
+<file>` names the final destination and moves the auto-named working file there. Every
+reply, `status` included, tells you the current path.
+
+**One recording, two replay drivers.** `semantic` (the default) is fast and moves no
+cursor — right for CI. `robot` glides the pointer to each target and clicks for real —
+the only mode worth filming, since a click with no pointer near it reads as broken. The
+recording is identical; only playback differs.
+
+Replay handles the timing traps a fixed `sleep` cannot: it **retries a step until it
+takes** (a menu item sits disabled for a moment after a modal dialog is dismissed) and
+**waits for a window a step opened** before moving on.
+
+The recording holds the navigation; assertions stay in the scenario. `detach-window-recorded.sh`
+shows the split — it replays one step at a time with `--from`/`--to` so it can check state
+*between* clicks, which a single end-to-end playback could not.
+
+## Capturing a feature for the help system
+
+```bash
+bridge.sh record start scenarios/recordings/my-feature.json
+# ... drive the feature by hand ...
+bridge.sh record stop
+
+bridge.sh replay scenarios/recordings/my-feature.json --shots /tmp/shots --shot-scale 0.5
+doc-scaffold.py scenarios/recordings/my-feature.json --shots /tmp/shots \
+                --target MyFeature --title "My Feature" --out page.xml
+```
+
+Then a person copies the captures worth keeping into
+`vcell-client/UserDocumentation/originalXML/topics/image/` under names that mean something,
+updates the `imgReference` targets, writes the prose, and adds a `<tocitem>` to `TOC.xml`.
+
+Three things this pipeline learned the hard way:
+
+- **`--shot-delay` matters.** `/idle` drains the EDT, but VCell fills many panels from
+  background tasks, so an idle EDT does not mean the pixels are final. Captured too early,
+  two different steps produce byte-identical images and the page documents the wrong screen.
+- **Scale the captures.** `DocumentCompiler` rejects any image over 500,000 bytes. At
+  `--shot-scale 0.5` a full window lands around 30KB; at full size it will not.
+- **The help build is skipped when it looks done.** The `build-documentation` profile
+  activates only when `target/classes/vcellDoc` is *missing*, so a plain rebuild silently
+  leaves doc changes untested. Remove that directory, or run `DocumentCompiler` directly.

tools/debug-bridge/bridge.sh

@@ -10,7 +10,11 @@
 #   health | windows | tree [maxDepth] | menus
 #   find    [--type T] [--name N] [--text T] [--contains S] [--limit N]
 #   props   <selector>          listeners <selector>
-#   shot [window]               log [lines]
+#   findrow <selector> <text> [--exact]   row number by displayed text (searches the WHOLE
+#                                         model, unlike tree's 25-row/100-row dump cap)
+#   findrow <selector> --apptype SPRINGSALAD   find an application by its type, not its
+#                                         name or position - both of which vary per model
+#   shot [window]               log [lines]      (shot takes ?scale/name/dir via replay)
 # Act:
 #   click <selector>            rclick <selector>
 #   settext <selector> <text> [--enter]
@@ -19,8 +23,16 @@
 #   trow <selector> <row> [col]   dtrow <selector> <row> [col]   rtrow <selector> <row> [col]
 #   menu "<Menu>Item[>Sub]" [window]
 #   highlight <selector> [ms]
+#   glide <selector> [ms]       move the real cursor there (for a watched replay)
+#   rbclick <selector> [glideMs] [row]  native press/release; unlike click it IS recordable
 #   iconify <selector> [true|false]   minimize/restore a window; reports what the OS did
 #   wbounds <selector> x y w h       move/resize a window
+# Record / replay:
+#   record start [file] [--no-bridge-actions] | stop [file] | status
+#            (flushed to disk each step; bridge-driven actions are recorded too unless
+#             --no-bridge-actions, so do scripted SETUP before starting the recording)
+#   replay <script.json> [--driver semantic|robot] [--speed N] [--max-delay MS]
+#                        [--from N] [--to N] [--shots DIR] [--shot-scale F]
 # Synchronize / assert (exit 0 on success, 1 on failure):
 #   wait   [find opts] [--state showing|enabled|gone] [--timeout MS] [--interval MS]
 #   assert [find opts] [--gone]
@@ -145,14 +157,51 @@ case "$cmd" in
                ${3:+--data-urlencode "column=$3"} | pretty ;;
   rrow)      get rightClickTreeRow --data-urlencode "path=$1" --data-urlencode "row=$2" | pretty ;;
   menu)      get menu --data-urlencode "path=$1" ${2:+--data-urlencode "window=$2"} | pretty ;;
+  findrow)
+    case "${2:-}" in
+      --apptype) get findRow --data-urlencode "path=$1" --data-urlencode "appType=$3" | pretty ;;
+      *) get findRow --data-urlencode "path=$1" \
+           $([ "${3:-}" = "--exact" ] && echo "--data-urlencode text=$2" || echo "--data-urlencode contains=$2") | pretty ;;
+    esac
+    ;;
   props)     get props --data-urlencode "path=$1" | pretty ;;
   listeners) get listeners --data-urlencode "path=$1" | pretty ;;
   highlight) get highlight --data-urlencode "path=$1" --data-urlencode "ms=${2:-2000}" | pretty ;;
+  glide)     get glide --data-urlencode "path=$1" --data-urlencode "ms=${2:-600}" | pretty ;;
+  rbclick)   get robotClick --data-urlencode "path=$1" --data-urlencode "glideMs=${2:-0}" \
+               ${3:+--data-urlencode "row=$3"} | pretty ;;
+
+  record)
+    action="${1:-status}"
+    case "$action" in
+      status) get record --data-urlencode "action=status" | pretty ;;
+      start)
+        # record start [file] [--no-bridge-actions]
+        cb=true
+        for a in "$@"; do [ "$a" = "--no-bridge-actions" ] && cb=false; done
+        f="$2"; [ "$f" = "--no-bridge-actions" ] && f=""
+        get record --data-urlencode "action=start" --data-urlencode "captureBridgeActions=$cb" \
+          ${f:+--data-urlencode "file=$f"} | pretty ;;
+      stop)  get record --data-urlencode "action=stop" ${2:+--data-urlencode "file=$2"} | pretty ;;
+      *) echo "record: expected start, stop or status" >&2; exit 2 ;;
+    esac
+    ;;
+
+  replay)
+    script="$1"
+    shift || true
+    PY=""
+    for candidate in python3 python py; do
+      if command -v "$candidate" >/dev/null 2>&1; then PY="$candidate"; break; fi
+    done
+    [ -n "$PY" ] || { echo "replay: need python3 on PATH" >&2; exit 2; }
+    "$PY" "$(dirname "$0")/replay.py" "$script" --port "$PORT" "$@"
+    ;;
   iconify)   get iconify --data-urlencode "path=$1" --data-urlencode "iconified=${2:-true}" | pretty ;;
   wbounds)   get windowBounds --data-urlencode "path=$1" --data-urlencode "x=$2" --data-urlencode "y=$3" --data-urlencode "w=$4" --data-urlencode "h=$5" | pretty ;;
 
   help|*)
-    sed -n '2,30p' "$0" | sed 's/^# \{0,1\}//'
+    sed -n '2,36p' "$0" | sed 's/^# \{0,1\}//'
     [ "$cmd" = help ] || exit 2
     ;;
 esac