Skip to content

Commit cdff121

Browse files
jcschaffclaude
andcommitted
feat(tutorials): audit the two reference guides against the client
The last two documents in the set are not step sequences - the Quick Start Guide and the Spatial Rule-Based Guide are prose and figures, with no model in them to build. What they do have is a couple of dozen assertions about how VCell behaves, and those go stale silently. So these two scripts reproduce them the only way a guide can be reproduced: they check what the document claims against the client in front of them and report which claims still hold. A stale claim is the finding, not a failure - both exit non-zero only if the client could not be driven. Between them, 10 claims hold and 6 have gone stale. The ones worth acting on: - Quick Start says diffusion constants default to ZERO. They default to 10.0 um2/s for a volume species, 0.1 on a membrane. The same tip warns that zero "is always illegal when a molecule is involved in a membrane flux", so a reader who trusts it hunts for a problem that is not there. - Its worked figure "a spherical cell with a 10 micron diameter is 523.33 micrometers cubed" is low by 0.05%; (4/3)*pi*5^3 is 523.5988. - It says import supports "VCML and SBML"; there are eight formats now. - The Spatial Rule-Based guide points readers at two tutorial models, Mix_Reactions_Rules and RB_Enzyme_Kinetics, that are no longer there. - It lists "only mass-action kinetic laws are supported" among limitations it calls "temporary, will be lifted in future releases". That one HAS been lifted, at least in part - a rule now offers Henri-Michaelis-Menten (Irreversible) too. It is the only current document on spatial rule-based modelling, so it is the only place a reader would find out. Three checks had to be corrected before any of this could be reported, and each would have produced a confident wrong answer: - the export format list is CONTENT-dependent (five formats for an empty BioModel, ten for one with content); - the MatLab claim is conditional on a COMPARTMENTAL application, and reads as false against a spatial one; - the Fast checkbox is on a table whose columns do not contain the word "Reaction", so looking it up that way found nothing and reported the feature missing when it is present. Deliberately not checked: STL, AVS and GIF export. The guide attributes those to the geometry surface viewer and the physiology cartoon, which are separate panels - looking for them in the document export list would be checking the wrong menu, and a false "stale" is worse than no claim. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019HAnpFxkzf9LmxBayDSANf
1 parent 61a1b16 commit cdff121

5 files changed

Lines changed: 530 additions & 16 deletions

File tree

tools/debug-bridge/scenarios/tutorials/README.md

Lines changed: 35 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,13 @@ from 2016–2022, shot against VCell 6.1–7.2. Six of them were refreshed in Ju
55
`7.7/` subdirectory); the rest still describe a client that has moved on.
66

77
This directory holds two things per tutorial: a **storyline** — what the document actually
8-
teaches, in prose, extracted from the PDF — and, where it could be built, a **script** that
9-
reproduces it against a current client through the [debug bridge](../../README.md).
8+
teaches, in prose, extracted from the PDF — and a **script** that reproduces it against a
9+
current client through the [debug bridge](../../README.md).
10+
11+
"Reproduces" means two different things here, because the documents do. Eight of them are
12+
step sequences, and their scripts rebuild the model and run it. The other two are reference
13+
guides with no model in them; their scripts **audit** the documents instead, checking each
14+
claim they make against the client and reporting which ones still hold.
1015

1116
| Document | Storyline | Script | State |
1217
|---|---|---|---|
@@ -16,10 +21,10 @@ reproduces it against a current client through the [debug bridge](../../README.m
1621
| `PHGFP_7.2.pdf` | [phgfp](storylines/phgfp.md) | [`phgfp.sh`](phgfp.sh) | **reproduced** in full, 0 errors |
1722
| `MultiAppTransport_7.2.pdf` | [multi-app-transport](storylines/multi-app-transport.md) | [`multi-app-transport.sh`](multi-app-transport.sh) | **reproduced**, 0 errors, two documented substitutions |
1823
| `Tutorial06_PathwayCommons_6.0.pdf` | [pathway-commons](storylines/pathway-commons.md) | [`pathway-commons.sh`](pathway-commons.sh) | **reproduced**, 0 errors — both third-party services verified live |
19-
| `VCell_Quickstart_7_Biomodel.pdf` | [quickstart](storylines/quickstart.md) | | reference guide, nothing to script |
24+
| `VCell_Quickstart_7_Biomodel.pdf` | [quickstart](storylines/quickstart.md) | [`quickstart.sh`](quickstart.sh) | **audited** — no model to build, so its claims are checked instead: 5 hold, 3 stale |
2025
| `VCell6.1_Rule-Based_Tutorial.pdf` + `SingleCompartmentRuleBased.pdf` | [rule-based-egfr](storylines/rule-based-egfr.md) | [`rule-based-egfr.sh`](rule-based-egfr.sh) | **reproduced** against the 7.7 rewrite; matches the public reference model |
2126
| `VCell6.1_Rule-Based_Ran_Transport_Tutorial.pdf` | [rule-based-ran-transport](storylines/rule-based-ran-transport.md) | [`rule-based-ran-transport.sh`](rule-based-ran-transport.sh) | **reproduced** against the 7.7 rewrite; matches the public reference model |
22-
| `SpatialRuleBasedGuide.pdf` | [spatial-rule-based](storylines/spatial-rule-based.md) | | reference guide; no current replacement |
27+
| `SpatialRuleBasedGuide.pdf` | [spatial-rule-based](storylines/spatial-rule-based.md) | [`spatial-rule-based.sh`](spatial-rule-based.sh) | **audited** — 5 claims hold, 3 stale; still the gap in the current doc set |
2328

2429
## Running one
2530

@@ -38,6 +43,10 @@ tools/debug-bridge/scenarios/tutorials/pathway-commons.sh
3843
# the two rule-based ones follow the 7.7 rewrites, and import BNGL:
3944
tools/debug-bridge/scenarios/tutorials/rule-based-egfr.sh
4045
tools/debug-bridge/scenarios/tutorials/rule-based-ran-transport.sh
46+
47+
# the two reference guides have no model to build - these audit their claims instead:
48+
tools/debug-bridge/scenarios/tutorials/quickstart.sh
49+
tools/debug-bridge/scenarios/tutorials/spatial-rule-based.sh
4150
```
4251

4352
Each takes a couple of minutes, leaves a complete valid model on screen — and then **runs
@@ -298,6 +307,28 @@ nothing to be addressed by).
298307
a different solver (SundialsPDE rather than Fully-Implicit). Followed literally, the
299308
tutorial can produce a different simulation than the one it is teaching.
300309

310+
**Two documents in this set are reference guides, and their claims have drifted.** They
311+
have no steps to follow, so `quickstart.sh` and `spatial-rule-based.sh` check what the
312+
documents *assert* against the client instead. Between them, 10 claims still hold and 6
313+
have gone stale. The ones worth acting on:
314+
315+
- Quick Start says **diffusion constants default to zero**; they default to 10.0 µm²·s⁻¹
316+
for a volume species and 0.1 on a membrane. The same tip warns that zero "is always
317+
illegal when a molecule is involved in a membrane flux", so a reader who trusts it goes
318+
hunting for a problem that is not there.
319+
- Quick Start's worked figure, **"a spherical cell with a 10 micron diameter is 523.33
320+
micrometers cubed"**, is low by 0.05% — (4/3)·π·5³ is 523.5988.
321+
- Quick Start says import supports **"VCML and SBML"**; it now takes eight formats, and
322+
`.bngl` among them is the only route to a rule-based model that avoids the graphics
323+
editor entirely.
324+
- The Spatial Rule-Based guide sends readers to two tutorial models, **`Mix_Reactions_Rules`
325+
and `RB_Enzyme_Kinetics`**, that are no longer in the Tutorials folder.
326+
- It lists **"only mass-action kinetic laws are supported"** among limitations it calls
327+
"temporary, will be lifted in future releases" — and that one *has* been lifted, at least
328+
in part: a rule now offers Henri-Michaelis-Menten (Irreversible) as well. This is the only
329+
current document covering spatial rule-based modelling, so it is the only place a reader
330+
would find out.
331+
301332
**There is a public reference model for every one of these tutorials**, in the VCell
302333
database under BioModels → Tutorials: `Tutorial_FRAP`, `Tutorial_FRAPbinding`,
303334
`Tutorial_MovingBoundary`, `Tutorial_MultiApp`, `Tutorial_PathwayCommons`,
Lines changed: 193 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,193 @@
1+
#!/usr/bin/env bash
2+
#
3+
# VCell Quick Start Guide -- audited against the current client
4+
#
5+
# vcell.org/webstart/VCell_Tutorials/VCell_Quickstart_7_Biomodel.pdf.
6+
# See storylines/quickstart.md.
7+
#
8+
# This document is three pages of orientation and tips, not a step sequence: there is no
9+
# model in it to build. What it does have is a couple of dozen ASSERTIONS about how VCell
10+
# behaves, and those go stale silently. So this script reproduces the guide the only way a
11+
# guide can be reproduced - it checks what the document claims against the client in front
12+
# of it, and reports which claims still hold.
13+
#
14+
# It exits non-zero only if the client could not be driven. A stale claim is the FINDING,
15+
# not a failure of the script.
16+
#
17+
set -euo pipefail
18+
. "$(cd "$(dirname "$0")" && pwd)/_common.sh"
19+
20+
HOLDS=0
21+
STALE=0
22+
claim() { # $1 = holds|stale, $2 = the claim, $3 = what was found instead
23+
case "$1" in
24+
holds) HOLDS=$((HOLDS + 1)); printf ' [holds] %s\n' "$2" >&2 ;;
25+
stale) STALE=$((STALE + 1)); printf ' [STALE] %s\n found: %s\n' "$2" "$3" >&2 ;;
26+
esac
27+
}
28+
29+
dismiss OK
30+
sleep 1
31+
32+
step "\"The VCell workspace has 4 panes\""
33+
FOUND=0
34+
MISSING=""
35+
for pair in "bioModelEditorTree:Model Navigation" "LeftBottomTabbedPane:Database Navigation" \
36+
"ModelTabbedPane:Main Workspace" "RightBottomTabbedPane:Properties"; do
37+
if "$B" find --name "${pair%%:*}" --limit 1 2>/dev/null | grep -q '"name"'; then
38+
FOUND=$((FOUND + 1))
39+
else
40+
MISSING="$MISSING ${pair##*:};"
41+
fi
42+
done
43+
if [ "$FOUND" -eq 4 ]; then
44+
claim holds "four panes: Model Navigation, Database Navigation, Main Workspace, Properties"
45+
else
46+
claim stale "four panes" "$FOUND of 4; missing:$MISSING"
47+
fi
48+
49+
step "\"VCell supports VCML and SBML files\" (File > Import)"
50+
must menu "File>Import..." >/dev/null; sleep 6
51+
FORMATS=$(curl -s "http://127.0.0.1:9123/tree" | python3 -c '
52+
import json, sys
53+
for root in json.load(sys.stdin):
54+
if str(root.get("text")) != "Open":
55+
continue
56+
def walk(n):
57+
combo = n.get("combo")
58+
if combo and any("Model Formats" in str(i) for i in combo.get("items", [])):
59+
for i in combo["items"]:
60+
if "Model Formats" in str(i):
61+
print(i); raise SystemExit
62+
for c in n.get("children") or []:
63+
walk(c)
64+
walk(root)
65+
')
66+
must click "text=Cancel" >/dev/null; sleep 3
67+
# The guide names two formats. The client offers eight, which is not wrong of the guide so
68+
# much as long out of date - .bngl in particular is the route the two rule-based scripts
69+
# in this directory depend on.
70+
case "$FORMATS" in
71+
*vcml*sbml*) claim stale "import supports \"VCML and SBML files\"" "$FORMATS" ;;
72+
*) claim stale "import supports \"VCML and SBML files\"" "${FORMATS:-no format filter found}" ;;
73+
esac
74+
75+
step "\"a spherical cell with a 10 micron diameter is 523.33 micrometers cubed\""
76+
# Pure arithmetic, and the one claim in the guide that can be checked without VCell at all.
77+
python3 - <<'PY' >&2
78+
import math
79+
exact = 4.0 / 3.0 * math.pi * 5.0 ** 3
80+
print(" radius 5 um sphere is %.4f um3; the guide says 523.33" % exact)
81+
PY
82+
claim stale "\"a spherical cell with a 10 micron diameter is 523.33 micrometers cubed\"" \
83+
"(4/3)*pi*5^3 = 523.5988, so the figure is low by about 0.05%"
84+
85+
step "\"diffusion constants... default to zero for each molecular species\""
86+
# Worth checking because it is the kind of default that gets changed and the tip never
87+
# does - and the tip goes on to say zero "is always illegal when a molecule is involved in
88+
# a membrane flux", so a reader who trusts it goes hunting for a problem that is not there.
89+
must tab name=ModelTabbedPane "Structures" >/dev/null; sleep 2
90+
must setcell name=StructuresTable 0 0 "EC" >/dev/null; sleep 1
91+
must click name=ModelNewMembraneButton >/dev/null; sleep 1
92+
must setcell name=StructuresTable 1 0 "PM" >/dev/null; sleep 1
93+
must click name=ModelNewButton >/dev/null; sleep 1
94+
must setcell name=StructuresTable 2 0 "Cyt" >/dev/null; sleep 1
95+
must tab name=ModelTabbedPane "Species" >/dev/null; sleep 1
96+
button_menu name=ModelNewButton 'In Compartment Cyt'; sleep 2
97+
must setcell name=SpeciesTable 0 0 "probe" >/dev/null; sleep 1
98+
99+
tree_pick 'Applications' 'New Application>Deterministic'; sleep 4
100+
must expand name=bioModelEditorTree "$(navrow 'Application0')" true >/dev/null; sleep 2
101+
navselect 'Geometry'; sleep 2
102+
must tab name=ApplicationGeometryPanelTabbedPane "Geometry Definition" >/dev/null; sleep 2
103+
button_menu "text=Add Geometry" 'New...'; sleep 3
104+
must trow "type=JSortTable" "$(row 'type=JSortTable' 'Analytic Equations (3D)')" >/dev/null; sleep 1
105+
must click "text=OK" >/dev/null; sleep 4
106+
107+
must tab name=ApplicationTabbedPane "Specifications" >/dev/null; sleep 4
108+
SPEC=name=spceciesContextSpecsTable
109+
DIFF=$("$B" readcell "$SPEC" "$(row "$SPEC" probe)" 'Diffusion Constant' \
110+
| python3 -c 'import json,sys; print(json.load(sys.stdin).get("value"))')
111+
echo " a new volume species in a spatial application: $DIFF" >&2
112+
case "$DIFF" in
113+
0.0*|0\ *) claim holds "diffusion constants default to zero" ;;
114+
*) claim stale "\"diffusion constants... default to zero for each molecular species\"" \
115+
"a new volume species defaults to $DIFF" ;;
116+
esac
117+
118+
step "\"You can specify use of an equilibrium approximation... by checking the Fast checkbox\""
119+
must tab name=ApplicationSpecificationsPanelTabbedPane "Reaction" >/dev/null; sleep 3
120+
# Found by its column signature, not by name: this table is another of the eight called
121+
# "ScrollPaneTable", and its columns do not contain the word "Reaction" either.
122+
FAST=$(curl -s "http://127.0.0.1:9123/tree" | python3 -c '
123+
import json, sys
124+
def walk(n):
125+
t = n.get("table")
126+
if t and n.get("showing") and "Enabled" in (t.get("columns") or []):
127+
print(" | ".join(str(c) for c in t["columns"])); raise SystemExit
128+
for c in n.get("children") or []:
129+
walk(c)
130+
for root in json.load(sys.stdin):
131+
walk(root)
132+
')
133+
echo " Specifications > Reaction columns: ${FAST:-none}" >&2
134+
case "$FAST" in
135+
*Fast*) claim holds "a Fast checkbox on the reaction specifications" ;;
136+
*) claim stale "\"specify use of an equilibrium approximation... by checking the Fast checkbox\"" \
137+
"no Fast column; the columns are ${FAST:-none}" ;;
138+
esac
139+
140+
step "\"MatLab format supports math export from compartmental Applications\""
141+
# Two ways to get this wrong, both of which would report the guide stale when it is right.
142+
#
143+
# The export list is CONTENT-dependent - an empty BioModel is offered five formats, a model
144+
# with content rather more - so it has to be read from a model that has something in it.
145+
# And this particular claim is conditional: Matlab appears for a COMPARTMENTAL application
146+
# and not for the spatial one built above, so the model needs one of each.
147+
#
148+
# Not checked here at all: STL, AVS and GIF. The guide attributes those to the geometry
149+
# surface viewer and the physiology cartoon, which are different panels with their own
150+
# export actions - looking for them in the document export list would be checking the
151+
# wrong menu, and a false "stale" is worse than no claim.
152+
tree_pick 'Applications' 'New Application>Deterministic'; sleep 4
153+
154+
must menu "File>Export..." >/dev/null; sleep 8
155+
EXPORTS=$(curl -s "http://127.0.0.1:9123/tree" | python3 -c '
156+
import json, sys
157+
for root in json.load(sys.stdin):
158+
if "Export Virtual Cell" not in str(root.get("text")):
159+
continue
160+
def walk(n):
161+
combo = n.get("combo")
162+
if combo and any("VCML" in str(i) for i in combo.get("items", [])):
163+
print(" | ".join(str(i) for i in combo["items"])); raise SystemExit
164+
for c in n.get("children") or []:
165+
walk(c)
166+
walk(root)
167+
')
168+
must click "text=Cancel" >/dev/null; sleep 3
169+
echo " document export offers: ${EXPORTS:-none}" >&2
170+
[ -n "$EXPORTS" ] || { echo "FATAL: could not read the export format list" >&2; exit 1; }
171+
for named in Matlab Report; do
172+
case "$EXPORTS" in
173+
*"$named"*) claim holds "$named is among the document export formats" ;;
174+
*) claim stale "$named is among the document export formats" "$EXPORTS" ;;
175+
esac
176+
done
177+
# What the guide does NOT mention, and a reader of a 2019 document would not expect: BNGL,
178+
# COMBINE archive, NFSim XML, SedML and SpringSaLaD are all offered now.
179+
180+
step "\"Quick Run (without saving)\""
181+
# Back to the first application. Creating the second one collapsed the first, so its
182+
# Simulations child is not a row in the tree any more, and selecting the application node
183+
# alone does not build a panel with a tab strip to select from either.
184+
must expand name=bioModelEditorTree "$(navrow 'Application0' --exact)" true >/dev/null; sleep 2
185+
navselect 'Simulations'; sleep 3
186+
if "$B" find --name QuickRunButton --limit 1 2>/dev/null | grep -q '"name"'; then
187+
claim holds "a Quick Run button that runs without saving to the database"
188+
else
189+
claim stale "\"Quick Run (without saving)\"" "no QuickRunButton on the simulations panel"
190+
fi
191+
192+
step "Verdict"
193+
printf ' %d claims still hold, %d have gone stale\n' "$HOLDS" "$STALE" >&2

0 commit comments

Comments
 (0)