Skip to content

Add KiminaProver (whole-proof generation prover)#1

Open
henryrobbins wants to merge 7 commits into
mainfrom
kimina
Open

Add KiminaProver (whole-proof generation prover)#1
henryrobbins wants to merge 7 commits into
mainfrom
kimina

Conversation

@henryrobbins

@henryrobbins henryrobbins commented Jun 22, 2026

Copy link
Copy Markdown
Owner

Adds Moonshot AI's Kimina-Prover as a registered prover (kimina): a whole-proof generation model we serve ourselves on GPU via vLLM. Structurally the Aristotle path (generate → splice → verify), with generation isolated behind a _generate seam so the splice/select/guard logic is fully testable offline.

Phase A — adapter (generation stubbed)

  • provers/_lean_splice.py — provider-agnostic, offset-preserving extract_theorems / splice_proof helpers (shared with the deferred BFS path). Bracket-aware so a := inside a signature isn't mistaken for the proof delimiter.
  • provers/kimina.pyKiminaProver + KiminaProverConfig. prove(): stage → extract sorry'd targets → _generate (test seam) → select by authoritative compile (first candidate that compiles, is sorry-free, and passes the StatementTracker signature guard wins) → diff. cost_usd=None; pass_k / samples_tried / per-target winning_index in metadata.
  • api.pykimina registry entry; the platform's agent backend doubles as Kimina's generation backend. No toolchain-gate change.
  • backends/modal.pyModalConfig.gpu, passed to Sandbox.create.

Phase B — GPU image + generation entrypoint

  • images/kimina/kimina_generate.py — one-shot vLLM entrypoint. Prompt format + sampling pinned to the model card recipe (temperature=0.6 top_p=0.95, n=pass_k). Extracts the proof body (tactic block after by) from the last complete fenced Lean block. vLLM/transformers imports deferred so helpers unit-test with no GPU.
  • images/kimina.Dockerfile — based on the official vLLM image, layering the standard Lean toolchain + warm Mathlib cache. Weights downloaded to a mounted Volume.
  • backends/modal.pyModalConfig.volumes → named Modal Volumes (persist weights / HF cache).
  • __main__.pybuild-kimina-image CLI (publishes a Modal image named kimina).

Phase C — partial (offline-testable wins)

  • Informal-problem context: extract_theorems captures each declaration's preceding doc comment (Theorem.docstring); threaded into generation as the problem field (per-target docstring, else task instructions).
  • Model-size knob: kimina:7b / kimina:1.5b registry variants; bare kimina keeps the 7B default.

Tests

tests/test_kimina_prover.py — pure helper coverage (incl. docstring extraction), pass@k selection + guard with a fake verifier, build_prompt/extract_proof_body units, problem-context threading, model-size variants, a Docker-marked end-to-end over the real shared verifier, and a slow Modal-gated real-GPU end-to-end. 54 offline tests pass; ruff + mypy clean.

Remaining (needs a real GPU / infra to build + verify)

  • Persistent vLLM server (avoid per-task model load).
  • Kimina Lean Server for fast in-sandbox pass@k pre-filtering before the authoritative compile.
  • GPU smoke test to close two open model-card questions: confirm the rendered chat template, and the distill's Mathlib revision vs our v4.28.0 pin.

🤖 Generated with Claude Code

henryrobbins and others added 5 commits June 22, 2026 18:05
…tubbed)

Add Kimina-Prover as a registered prover (`kimina`): a whole-proof generation
model we serve ourselves on GPU. Structurally the Aristotle path (generate ->
splice -> verify), with generation isolated behind a `_generate` seam so the
splice/select/guard logic is fully testable offline.

- new provers/_lean_splice.py: provider-agnostic, offset-preserving
  extract_theorems / splice_proof helpers (shared with the deferred BFS path).
- new provers/kimina.py: KiminaProver + KiminaProverConfig. pass@k selection by
  authoritative compile (first candidate that compiles sorry-free and leaves the
  signature intact wins); StatementTracker guard against signature drift;
  `_generate` runs a one-shot kimina_generate.py in a GPU backend (Phase B image).
- api.py: registry entry + wire the agent backend as Kimina's generation backend.
- backends/modal.py: ModalConfig.gpu, passed to Sandbox.create.
- tests/test_kimina_prover.py: pure helper tests, pass@k selection + guard with a
  fake verifier (offline), and a Docker-marked end-to-end over the real verifier.
- docs: a Kimina prover page + API reference.

Phase B (GPU image + kimina_generate.py + build-kimina-image CLI) and Phase C
(persistent vLLM server, Lean-server pre-filtering) remain.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire the real GPU generation path behind KiminaProver._generate.

- images/kimina/kimina_generate.py: one-shot vLLM generation entrypoint baked into
  the image. Prompt format + sampling pinned to the Kimina-Prover-Preview-Distill-7B
  model card (system prompt, # Problem / # Formal statement template,
  temperature=0.6 top_p=0.95, n=pass_k). Extracts the proof body (tactic block after
  `by`) from the last complete fenced Lean block, ready to splice. vLLM/transformers
  imports are deferred so the prompt/extraction helpers unit-test with no GPU.
- images/kimina.Dockerfile: based on the official vLLM image (known-good CUDA/torch/
  vLLM stack), layering the platform's standard Lean toolchain + warm Mathlib cache.
  Weights are not baked; vLLM downloads to HF_HOME, persisted by a mounted Volume.
- backends/modal.py: ModalConfig.volumes -> named Modal Volumes mounted in the
  Sandbox (persist weights / HF cache across runs).
- __main__.py: `build-kimina-image` CLI (publishes a Modal image named "kimina");
  refactored the Modal build into a shared _publish_modal_image helper.
- tests: offline unit tests for build_prompt / extract_proof_body, plus a slow,
  Modal-gated real-GPU end-to-end asserting the shared verifier confirms the proof.
- docs: build instructions, pinned recipe, and volume usage on the Kimina page.

Phase C (persistent vLLM server, Lean-server pre-filtering, model-size knob,
informal-problem context) remains. Open: pin/smoke-test the exact chat template and
the distill's Mathlib revision vs our v4.28.0 pin on a real GPU run.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two offline-testable Phase C wins; the persistent vLLM server and Kimina Lean-server
pre-filtering remain (they need infra not testable here).

- _lean_splice.py: extract_theorems now captures each declaration's preceding
  doc/block comment (Theorem.docstring) -- the informal problem statement in
  miniF2F-style files.
- kimina.py: thread that informal context into generation as the `problem` field
  (per-target docstring, else the task instructions). _generate now takes one
  {name, statement, problem} payload per target; kimina_generate.py already consumes
  `problem`.
- api.py: `kimina:7b` / `kimina:1.5b` registry variants select a distilled checkpoint
  without hand-wiring config; bare `kimina` keeps the 7B default.
- tests: docstring extraction, problem-context threading (+ instructions fallback),
  and the model-size variants.
- docs: informal context, model-size variants, and a roadmap note for the remaining
  optimizations.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
From the first real Modal GPU run: the GPU sat idle (~0% util) during a ~75s cold
weight download (network-bound), and generation produced 0 usable candidates with no
visibility into why (artifact pull-back also failed).

- backends/modal.py: ModalConfig.warm_lean (default True). Skip the `lake build` warm
  step on pure-generation GPU sandboxes -- it's CPU Lean work billed at GPU rates and
  useless for vLLM. The e2e generation backend sets warm_lean=False.
- provers/kimina.py: _generate now checks the generation command's exit code and
  raises with the stderr/stdout tail when no candidates come back, instead of silently
  yielding nothing.
- images/kimina/kimina_generate.py: per-completion stderr diagnostics (finish_reason,
  output length, raw tail) on extraction misses -- finish_reason="length" flags a
  truncated reasoning trace. stderr is pulled back reliably even when the workdir pull
  fails.

Weight download is amortized by the kimina-hf-cache volume (run #2 skips it); GPU
class does not affect startup (idle during download), so no hardware change needed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The first real GPU run revealed the model works (emits a valid `ring` proof) but
extract_proof_body returned 0 candidates. Cause: the <think> reasoning trace contains
its own draft code fences plus an occasional stray, unbalanced fence, which misaligns
whole-output fence pairing -- the real proof block (after </think>) gets consumed as a
closing fence and is never captured (and a draft proof could be grabbed instead).

Fix: anchor extraction on the region after the last </think> (the final answer), only
falling back to the whole text if that yields nothing. Added an offline regression
test for the real shape.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
henryrobbins and others added 2 commits June 22, 2026 20:24
build_prover gave Kimina the shared agent_backend, which carries the CPU
DEFAULT_IMAGE (no /opt/kimina/kimina_generate.py) and no GPU. Build a
dedicated ModalBackend pinned to the published kimina image + requested
GPU + a persistent HF-cache volume instead.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The one-shot prover prompted the model with only the isolated theorem
signature -- e.g. `Nonempty (MILPReformulation P6.a.formulation
P6.b.formulation)` -- and an empty informal field. Every supporting
definition (the inlined `Common` structures, both formulation defs)
already lives in the task .lean file but was discarded, so the model saw
only type *names* and could not know the shape of the witness to build.
It fell back to trivial guesses.

Capture the source preceding the target declaration as `Theorem.preamble`
(imports + supporting defs), forward it as a new `context` payload field,
and render it ahead of the formal statement in `build_prompt` so the model
sees a self-contained block where every type the goal names is defined.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@henryrobbins henryrobbins force-pushed the main branch 4 times, most recently from a715ac7 to 832e585 Compare June 29, 2026 16:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant