SimpleTES

Test-time compute, spent where it actually matters.

SimpleTES (Simple Test-time Evaluation-driven Scaling) scales the propose → evaluate → refine loop for scientific discovery. It combines parallel exploration, feedback-driven refinement, and local selection. A single open-source gpt-oss model reaches state-of-the-art results across 28 open-ended problems in five scientific domains, beating both frontier models and tuned optimization pipelines.

Updates

• 2026-04 — Released the SimpleTES technical report on arXiv and the public codebase.

Highlight Results

Domain	Representative problems
Quantum compilation	Superconducting qubit routing and zoned neutral-atom compilation
Astrodynamics	Gravity-assist trajectory design
Scientific algorithms	LASSO path solving, whole-brain forecasting, and single-cell RNA denoising
AI foundations	GPU kernel optimization and scaling-law discovery
Mathematics discovery	Extremal analysis and combinatorial construction

• Task catalogue and design guide: datasets/README.md.
• Released result artifacts: best_results/README.md.
• Case studies — what each task's seed program evolved into, with side-by-side animations: assets/case_study/README.md.

How It Works

A fixed evaluator budget is spent across four levers:

Lever	Knob	Controls
C	--num-chains	Parallel exploration — try genuinely different directions
L	implied by total budget	Feedback-driven refinement depth per chain
K	--k-candidates	Local best-of-K — avoid committing weak candidates
Φ	--selector	History-to-prompt policy — what past evidence shapes the next attempt

Recommended configuration

Use a stronger model with smaller C, L, and K to reach good results faster.

Each trajectory keeps a history of (candidate, score, metadata). One step: select history, build prompt, ask for K candidates, evaluate in isolated subprocesses, commit the best.

Available selectors (uv run python main.py --list-policies):

Selector	Style	Best for
balance (default)	Stratified sampling	Robust default; low-config exploration
puct	PUCT scoring	Tree-search flavored selection
rpucg	DAG-aware, γ-decay	Paper-style; strongest single selector
llm_elite	Bounded elite pool	LLM-managed per-chain population
llm_puct / llm_rpucg	Hybrid prefilter + LLM	Best for noisy chains / rich DAG histories

Quickstart

Configure the model API before launching a run. Set credentials for any LiteLLM-supported provider:

export GEMINI_API_KEY=...      # or OPENAI_API_KEY / ANTHROPIC_API_KEY / ...

For a custom endpoint, enter the API base when running main_wizard.py or pass --api-base to the CLI. Prefer environment variables for API keys so secrets do not end up in shell history or committed configuration.

Then install Python ≥ 3.11 dependencies with uv:

uv sync
uv sync --extra vllm        # optional: vLLM token-forcing backend

Or pip install -e ..

Interactive launcher (discovers tasks, prompts for model / budget / selector, prints or runs the command):

uv run python main_wizard.py

Direct CLI:

uv run python main.py \
  --init-program  datasets/circle_packing/circle_packing_26/init_program.py \
  --evaluator     datasets/circle_packing/circle_packing_26/evaluator.py \
  --instruction   datasets/circle_packing/circle_packing_26/circle_packing_26.txt \
  --model         gemini/gemini-2.0-flash \
  --selector      rpucg \
  --max-generations 50

Resume from an instance directory:

uv run python main.py --resume checkpoints/<date>/instance-<id>

All flags: uv run python main.py --help.

Configuration

📋 Most-tuned flags — click to expand

Goal	Flag
Total search budget	--max-generations
More directions	--num-chains
Less myopic local picks	--k-candidates
Change history-to-prompt strategy	--selector
Per-chain in-flight cap (concurrency)	--backpressure-multiplier
Split throughput knobs	--gen-concurrency, --eval-concurrency
Early stop when score reached	--early-stop-score
Inspirations per prompt (or a sampled range)	--num-inspirations / --min-inspirations-cnt / --max-inspirations-cnt
Switch LLM backend	--llm-backend litellm (default) or --llm-backend vllm_token_forcing
Use a task-local Python env	--eval-venv <path> (auto-detected for datasets/<family>/venv/)
Skip the 1-token LLM ping at startup	--skip-preflight

🎚️ Selector-specific flags — click to expand

Selector	Flag
balance	--exploitation-ratio / --exploration-ratio / --elite-ratio
puct	--puct-c
rpucg	--rpucg-gamma
llm_elite / llm_puct / llm_rpucg	--llm-policy-model / --llm-policy-api-base / --llm-policy-api-key / --llm-policy-pool-size

💾 Checkpoint & resume — click to expand

Checkpoints land under --output-path (default checkpoints/) every --log-interval evaluations. Each run gets a <date>/instance-<id>/ directory. Resume with --resume pointed at that instance directory:

uv run python main.py --resume checkpoints/<date>/instance-<id>

--save-llm-io keeps full LLM input/output (large files). --gzip compresses checkpoint nodes.

Build Your Own Task

Task implementations are organized by family under datasets/. A new task is three files:

my_family/
  my_task/
    init_program.{py|cpp|rs|...}      # seed; mark the evolved region with EVOLVE-BLOCK
    evaluator.py                      # def evaluate(filepath) -> {"combined_score": ..., ...}
    my_task.txt                       # instruction shown to the model

Drop the directory under datasets/ and main_wizard.py picks it up.

→ Catalogue + design guide: datasets/README.md.

Contributing

Contributions are welcome.

• New tasks: follow datasets/README.md and open a PR.
• Code: fork, add tests under tests/, run uv run pytest, open a PR with a benchmark comparison.
• Bugs / features: GitHub Issues.

[Report a Bug] | [Suggest a Feature] | [Open a PR] | [Add a Task]

Troubleshooting

The most common issues and how to resolve them are listed below.

Symptom	Action
LLM preflight failed	Check provider credentials, model string, and API base. Use --skip-preflight to bypass while the backend is still warming up.
No checkpoints found on resume	Pass the instance directory (e.g. checkpoints/2026-05-24/instance-0), not its parent date directory.
Task complains about missing files	uv run python scripts/prepare_task.py --list to see what's available, then --task <family> to fetch / build.
Evaluations error on imports	Pin the task-local venv with --eval-venv <path>; SimpleTES auto-detects datasets/<family>/venv/ when present.
Hitting rate limits	Lower --gen-concurrency, raise --retry, or switch to a lower-latency model.
Evaluations time out	Raise --eval-timeout for slow compilers / simulators. The task-level default lives in each evaluator as TIMEOUT_SECONDS and can be overridden per-evaluation via EVALUATOR_TIMEOUT_SECONDS.
GPU-kernel tasks (gpukernel) hang	Make sure the compiler server is running first — see the family README.md for the launch command.
fcntl import error (Windows)	The registry script scripts/evolve_db_registry.py is POSIX-only by design. Other tasks run fine on Windows.

Community

Join the SimpleTES community to discuss usage, share research progress, and send feedback. Scan the QR code below to join the chat group:

Citation

@article{simpletes2026,
  title   = {Evaluation-driven Scaling for Scientific Discovery},
  author  = {WILL Team},
  journal = {arXiv preprint arXiv:2604.19341},
  year    = {2026},
  url     = {https://arxiv.org/abs/2604.19341}
}

License

Released under GNU AGPL-3.0-or-later, © 2026 WILL.

• Research and local use — allowed.
• Programs discovered by SimpleTES — not automatically AGPL just because SimpleTES found them.
• Modifying the framework and distributing it — the derivative framework stays AGPL.
• Exposing a modified version as a network service — you must provide source under AGPL terms.