Skip to content

Add runtime benchmark baseline before Msg redesign#1023

Open
emil14 wants to merge 11 commits intomainfrom
codex/runtime-benchmarks-baseline
Open

Add runtime benchmark baseline before Msg redesign#1023
emil14 wants to merge 11 commits intomainfrom
codex/runtime-benchmarks-baseline

Conversation

@emil14
Copy link
Copy Markdown
Collaborator

@emil14 emil14 commented Feb 14, 2026

Summary

This PR adds runtime benchmark baselines that should land in main before merging #1004 (Redesign runtime Msg).

It addresses review comment: #1004 (comment)

What is included

  • Reworked runtime message-passing benchmark to be true e2e runtime execution:
    • builds cmd/neva once
    • compiles benchmark program once
    • times only execution of compiled output binary
  • Increased message-passing workload from 1000 to 100000 to reduce startup/process noise in measurements
  • Added focused internal/runtime microbenchmarks for stable runtime hotspots:
    • list iteration
    • dict lookup
    • list equality
    • struct field lookup
    • single-port send/receive round-trip

Why this shape

  • e2e benchmark gives release-level runtime signal
  • microbenchmarks give stable lower-level signal for runtime internals
  • together they provide a baseline for comparing Redesign runtime Msg #1004 and future runtime changes

Validation

  • golangci-lint run ./...
  • go test ./internal/runtime/...
  • go test ./benchmarks/message_passing -run=^$ -bench=BenchmarkMessagePassingE2E -benchtime=1x

Copy link
Copy Markdown
Collaborator Author

@emil14 emil14 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One new finding from a fresh pass on the current branch: benchmarks/message_passing/bench_test.go:35 uses ReportAllocs() even though this suite measures an external compiled binary. In this setup the reported allocations are only from the Go benchmark harness, not from the spawned Neva program, and the current results already show a nearly flat ~11 KB/op across many very different benchmark cases. That makes allocs/op here actively misleading as a runtime signal. I would drop ReportAllocs() from the e2e suite and keep allocation tracking in the lower-level runtime benchmarks instead.

@emil14
Copy link
Copy Markdown
Collaborator Author

emil14 commented Mar 22, 2026

Fresh pass is now in 2af5ff4f.

What changed:

  • merged current main into this branch and resolved the AGENTS.md / pkg/e2e drift
  • moved the harness to benchmarks/bench_test.go and flattened fixtures into benchmarks/simple/** + benchmarks/complex/**
  • removed misleading e2e allocation reporting and switched the timed loop to io.Discard + Run()
  • reused shared setup from pkg/e2e and added benchmark-specific helpers there
  • added runtime-benchmarks GitHub Actions workflow that uploads raw benchmark artifacts on PRs / main / releases

Validation:

  • go run github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.5.0 run ./benchmarks ./pkg/e2e
  • go test ./pkg/e2e -count=1
  • go test ./benchmarks -run=^$ -bench 'BenchmarkRuntimeE2E/simple_message_passing_int$' -benchtime=1x -count=1
  • go test ./benchmarks -run=^$ -bench BenchmarkRuntimeE2E -benchtime=1x -count=1

Note: the full suite is still heavy (~293s locally on Apple M1 Pro), so the workflow is artifact-only for now rather than a hard perf gate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant