Add KiminaProver (whole-proof generation prover)#1
Open
henryrobbins wants to merge 7 commits into
Open
Conversation
…tubbed) Add Kimina-Prover as a registered prover (`kimina`): a whole-proof generation model we serve ourselves on GPU. Structurally the Aristotle path (generate -> splice -> verify), with generation isolated behind a `_generate` seam so the splice/select/guard logic is fully testable offline. - new provers/_lean_splice.py: provider-agnostic, offset-preserving extract_theorems / splice_proof helpers (shared with the deferred BFS path). - new provers/kimina.py: KiminaProver + KiminaProverConfig. pass@k selection by authoritative compile (first candidate that compiles sorry-free and leaves the signature intact wins); StatementTracker guard against signature drift; `_generate` runs a one-shot kimina_generate.py in a GPU backend (Phase B image). - api.py: registry entry + wire the agent backend as Kimina's generation backend. - backends/modal.py: ModalConfig.gpu, passed to Sandbox.create. - tests/test_kimina_prover.py: pure helper tests, pass@k selection + guard with a fake verifier (offline), and a Docker-marked end-to-end over the real verifier. - docs: a Kimina prover page + API reference. Phase B (GPU image + kimina_generate.py + build-kimina-image CLI) and Phase C (persistent vLLM server, Lean-server pre-filtering) remain. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire the real GPU generation path behind KiminaProver._generate. - images/kimina/kimina_generate.py: one-shot vLLM generation entrypoint baked into the image. Prompt format + sampling pinned to the Kimina-Prover-Preview-Distill-7B model card (system prompt, # Problem / # Formal statement template, temperature=0.6 top_p=0.95, n=pass_k). Extracts the proof body (tactic block after `by`) from the last complete fenced Lean block, ready to splice. vLLM/transformers imports are deferred so the prompt/extraction helpers unit-test with no GPU. - images/kimina.Dockerfile: based on the official vLLM image (known-good CUDA/torch/ vLLM stack), layering the platform's standard Lean toolchain + warm Mathlib cache. Weights are not baked; vLLM downloads to HF_HOME, persisted by a mounted Volume. - backends/modal.py: ModalConfig.volumes -> named Modal Volumes mounted in the Sandbox (persist weights / HF cache across runs). - __main__.py: `build-kimina-image` CLI (publishes a Modal image named "kimina"); refactored the Modal build into a shared _publish_modal_image helper. - tests: offline unit tests for build_prompt / extract_proof_body, plus a slow, Modal-gated real-GPU end-to-end asserting the shared verifier confirms the proof. - docs: build instructions, pinned recipe, and volume usage on the Kimina page. Phase C (persistent vLLM server, Lean-server pre-filtering, model-size knob, informal-problem context) remains. Open: pin/smoke-test the exact chat template and the distill's Mathlib revision vs our v4.28.0 pin on a real GPU run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two offline-testable Phase C wins; the persistent vLLM server and Kimina Lean-server
pre-filtering remain (they need infra not testable here).
- _lean_splice.py: extract_theorems now captures each declaration's preceding
doc/block comment (Theorem.docstring) -- the informal problem statement in
miniF2F-style files.
- kimina.py: thread that informal context into generation as the `problem` field
(per-target docstring, else the task instructions). _generate now takes one
{name, statement, problem} payload per target; kimina_generate.py already consumes
`problem`.
- api.py: `kimina:7b` / `kimina:1.5b` registry variants select a distilled checkpoint
without hand-wiring config; bare `kimina` keeps the 7B default.
- tests: docstring extraction, problem-context threading (+ instructions fallback),
and the model-size variants.
- docs: informal context, model-size variants, and a roadmap note for the remaining
optimizations.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
From the first real Modal GPU run: the GPU sat idle (~0% util) during a ~75s cold weight download (network-bound), and generation produced 0 usable candidates with no visibility into why (artifact pull-back also failed). - backends/modal.py: ModalConfig.warm_lean (default True). Skip the `lake build` warm step on pure-generation GPU sandboxes -- it's CPU Lean work billed at GPU rates and useless for vLLM. The e2e generation backend sets warm_lean=False. - provers/kimina.py: _generate now checks the generation command's exit code and raises with the stderr/stdout tail when no candidates come back, instead of silently yielding nothing. - images/kimina/kimina_generate.py: per-completion stderr diagnostics (finish_reason, output length, raw tail) on extraction misses -- finish_reason="length" flags a truncated reasoning trace. stderr is pulled back reliably even when the workdir pull fails. Weight download is amortized by the kimina-hf-cache volume (run #2 skips it); GPU class does not affect startup (idle during download), so no hardware change needed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The first real GPU run revealed the model works (emits a valid `ring` proof) but extract_proof_body returned 0 candidates. Cause: the <think> reasoning trace contains its own draft code fences plus an occasional stray, unbalanced fence, which misaligns whole-output fence pairing -- the real proof block (after </think>) gets consumed as a closing fence and is never captured (and a draft proof could be grabbed instead). Fix: anchor extraction on the region after the last </think> (the final answer), only falling back to the whole text if that yields nothing. Added an offline regression test for the real shape. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
build_prover gave Kimina the shared agent_backend, which carries the CPU DEFAULT_IMAGE (no /opt/kimina/kimina_generate.py) and no GPU. Build a dedicated ModalBackend pinned to the published kimina image + requested GPU + a persistent HF-cache volume instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The one-shot prover prompted the model with only the isolated theorem signature -- e.g. `Nonempty (MILPReformulation P6.a.formulation P6.b.formulation)` -- and an empty informal field. Every supporting definition (the inlined `Common` structures, both formulation defs) already lives in the task .lean file but was discarded, so the model saw only type *names* and could not know the shape of the witness to build. It fell back to trivial guesses. Capture the source preceding the target declaration as `Theorem.preamble` (imports + supporting defs), forward it as a new `context` payload field, and render it ahead of the formal statement in `build_prompt` so the model sees a self-contained block where every type the goal names is defined. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
a715ac7 to
832e585
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Moonshot AI's Kimina-Prover as a registered prover (
kimina): a whole-proof generation model we serve ourselves on GPU via vLLM. Structurally the Aristotle path (generate → splice → verify), with generation isolated behind a_generateseam so the splice/select/guard logic is fully testable offline.Phase A — adapter (generation stubbed)
provers/_lean_splice.py— provider-agnostic, offset-preservingextract_theorems/splice_proofhelpers (shared with the deferred BFS path). Bracket-aware so a:=inside a signature isn't mistaken for the proof delimiter.provers/kimina.py—KiminaProver+KiminaProverConfig.prove(): stage → extract sorry'd targets →_generate(test seam) → select by authoritative compile (first candidate that compiles, issorry-free, and passes theStatementTrackersignature guard wins) → diff.cost_usd=None;pass_k/samples_tried/ per-targetwinning_indexin metadata.api.py—kiminaregistry entry; the platform's agent backend doubles as Kimina's generation backend. No toolchain-gate change.backends/modal.py—ModalConfig.gpu, passed toSandbox.create.Phase B — GPU image + generation entrypoint
images/kimina/kimina_generate.py— one-shot vLLM entrypoint. Prompt format + sampling pinned to the model card recipe (temperature=0.6 top_p=0.95,n=pass_k). Extracts the proof body (tactic block afterby) from the last complete fenced Lean block. vLLM/transformers imports deferred so helpers unit-test with no GPU.images/kimina.Dockerfile— based on the official vLLM image, layering the standard Lean toolchain + warm Mathlib cache. Weights downloaded to a mounted Volume.backends/modal.py—ModalConfig.volumes→ named Modal Volumes (persist weights / HF cache).__main__.py—build-kimina-imageCLI (publishes a Modal image namedkimina).Phase C — partial (offline-testable wins)
extract_theoremscaptures each declaration's preceding doc comment (Theorem.docstring); threaded into generation as theproblemfield (per-target docstring, else taskinstructions).kimina:7b/kimina:1.5bregistry variants; barekiminakeeps the 7B default.Tests
tests/test_kimina_prover.py— pure helper coverage (incl. docstring extraction), pass@k selection + guard with a fake verifier,build_prompt/extract_proof_bodyunits, problem-context threading, model-size variants, a Docker-marked end-to-end over the real shared verifier, and a slow Modal-gated real-GPU end-to-end. 54 offline tests pass; ruff + mypy clean.Remaining (needs a real GPU / infra to build + verify)
v4.28.0pin.🤖 Generated with Claude Code