Skip to content

perf(lez): run guests in-process on the node via the memoized image cache - #872

Merged
moudyellaz merged 2 commits into
devfrom
moudy/prod-image-cache
Sep 10, 2026
Merged

moudyellaz merged 2 commits into
devfrom
moudy/prod-image-cache

Conversation

@moudyellaz

@moudyellaz moudyellaz commented Sep 9, 2026 •

Copy link
Copy Markdown
Collaborator

🎯 Purpose

The sequencer and indexer build without prove, so lee/state_machine/src/program/mod.rs:122 resolves to risc0's ExternalProver: a fresh r0vm subprocess per guest execution, an r0vm --version probe per call, and an ELF-to-memory-image rebuild every time. With prove the same call site uses image_cache::execute, which runs in-process against a memoized image. The two are selected by a hard cfg, and only unit-tests (via --all-features) has ever exercised the fast one.

Measured on one machine, RISC0_DEV_MODE=1, release:

subprocess in-process
storage replay test (~1300 executions) 45.96s 3.19s
marginal block, one charged tx ~690ms ~50ms
tps_test, 1500 charged txs failed, 0/1500 confirmed in the 187.5s window, 692s total 65s, 23 TPS

The last row is the headline: the node as it ships cannot reach the repo's own 8 TPS target, and reaches 23 TPS with this change.

Binary size is unchanged (40,707,392 vs 40,707,600 bytes) because only the executor is reachable and the proving circuits are dead-stripped. The cost is build time: 77 additional crates, seven of them native.

⚙️ Approach

  • Add a prove feature to chain_state, sequencer_core, sequencer_service, indexer_core, indexer_service, each forwarding to lee/prove, reusing the existing storage/prove
  • Default FEATURES to prove in both service Dockerfiles
  • FEATURES=prove,mdns for the sequencer_service-mdns image, which passed FEATURES=mdns and would otherwise have overridden the default and silently stayed on the subprocess path

🧪 How to Test

Confirm the executor actually changes, not just the feature:

cargo tree -p sequencer_service -e normal -f "{p} :: {f}" | grep risc0-zkvm
cargo tree -p sequencer_service --features prove -e normal -f "{p} :: {f}" | grep risc0-zkvm

client,std vs client,prove,std. Then reproduce the throughput difference:

cargo test -p integration_tests --release --test tps -- --nocapture
cargo test -p integration_tests --release --features prove --test tps -- --nocapture

🔜 Future Work

  • Integration tests run the subprocess executor too, because Cargo.toml:105 pins wallet-ffi with default-features = false. Wiring prove into the 35 CI shards was tried and abandoned: enabling it also flips default_prover() to LocalProver, whose dev-mode path unwraps at dev_mode.rs:167 instead of returning an error, so auth_transfer::private::ppt_cant_chain_call_faucet panics where it used to get a clean Err. The speedup also looks small there, since integration tests wait on block timers rather than being executor-bound. Neither issue affects this PR: the services never call default_prover().
  • Drop the r0vm binary and RISC0_SERVER_PATH from both images once nothing can fall back to ExternalProver.

📋 PR Completion Checklist

  • Complete PR description
  • Implement the core functionality
  • Add/update tests
  • Add/update documentation and inline comments

@moudyellaz moudyellaz added the priority:high High priority label Sep 10, 2026
@moudyellaz
moudyellaz requested a review from Pravdyvy September 10, 2026 09:14

@Pravdyvy Pravdyvy left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@moudyellaz
moudyellaz merged commit 35dedce into dev Sep 10, 2026
50 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority:high High priority

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants