feat: expose resume evidence states

Author: cheeweijie

.github/workflows/instavar-voice-contract.yml

@@ -17,7 +17,7 @@ jobs:
         uses: actions/checkout@v5
         with:
           repository: instavar/instavar-voice-evaluation
-          ref: 8c0fb66a592c73f801a289aabd242e03a6849115
+          ref: 29c38cfd86b889abc8b79df063c817dd8f684903
           path: .instavar-voice-evaluation
       - uses: actions/setup-python@v6
         with:
@@ -32,6 +32,10 @@ jobs:
         env:
           PYTHONPATH: .instavar-voice-evaluation
         run: python -c "from inspect import signature; from instavar_voice_lab.content_faithfulness import CONTENT_FAITHFULNESS_SCHEMA_VERSION as v, build_content_faithfulness_report as b; assert v == 'instavar_voice_content_faithfulness/v1'; assert {'generation_plan', 'reference_catalog', 'reference_assignment_plan', 'speaker_references', 'instruction_ngram_size', 'minimum_instruction_ngram_hits'} <= set(signature(b).parameters)"
+      - name: Verify resume evidence API
+        env:
+          PYTHONPATH: .instavar-voice-evaluation
+        run: python -c "from inspect import signature; from instavar_voice_lab.resume import LIVE_CONDITIONING_SCHEMA_VERSION as v, build_resume_run_receipt as b, compare_resume_artifacts as c; assert v == '1.1.0'; assert 'identity_artifacts' in signature(b).parameters; assert 'output_path' in signature(c).parameters"
       - name: Validate historical run evidence
         run: python .instavar-voice-evaluation/main.py validate historical historical-run.json
       - name: Validate executable lifecycle backend

README.md

@@ -252,6 +252,20 @@ Each atomic `resume_epoch_NNNNNN` contains adapter bytes, optimizer, scheduler,
 AMP scaler when enabled, Python, NumPy, Torch, and CUDA RNG state, completed
 epoch and step, plus CV early-stop history.
 
+Future checkpoints also bind separate `optimizer-state.pt` and
+`scheduler-state.pt` files. Together with `adapter_model.safetensors`,
+`training-state.json`, and `runtime-state.pt`, they expose the five independent
+file roles required by Instavar Voice evaluator 0.45. The combined runtime file
+remains the guarded loader source, so older checkpoints without the decomposed
+copies still resume under their original sidecar authority.
+
+`evaluator_lora_artifact_paths(...)` rechecks the sidecar-bound live bytes and
+rejects missing decomposed state, ambiguous model files, and cross-role
+hardlinks. This instrumentation does not upgrade existing training or quality
+evidence. A fresh run pair must bind live Base, lineage, controls, and initial
+state through schema 1.1 receipts before outcome inspection. See
+[`reports/resume-evaluator-045-instrumentation-2026-08-14.md`](reports/resume-evaluator-045-instrumentation-2026-08-14.md).
+
 The selected directory must be the newest owned guarded checkpoint. Changed
 bytes, changed inputs, terminal symlinks, unowned numeric siblings, a completed
 `max_epoch`, or an already-triggered early-stop target fail before adapter or
@@ -746,7 +760,7 @@ Tools in this repo are Apache-2.0 licensed. CosyVoice itself is under the [CosyV
 
 ## Instavar Voice conformance
 
-[`instavar-voice-capabilities.json`](instavar-voice-capabilities.json) records the validated PyTorch adapter and merged-weight vLLM paths, while keeping direct vLLM LoRA loading explicitly unsupported. It also freezes the shared objective and blinded-listening criteria that remain necessary before a perceptual promotion decision. CI validates the manifest against the pinned public [Instavar Voice evaluation contract](https://github.com/instavar/instavar-voice-evaluation). New lifecycle runs should use evaluator commit `8c0fb66a592c73f801a289aabd242e03a6849115` or a deliberately reviewed successor so POSIX stage timeouts clean the complete process group. This does not retroactively upgrade earlier run evidence.
+[`instavar-voice-capabilities.json`](instavar-voice-capabilities.json) records the validated PyTorch adapter and merged-weight vLLM paths, while keeping direct vLLM LoRA loading explicitly unsupported. It also freezes the shared objective and blinded-listening criteria that remain necessary before a perceptual promotion decision. CI validates the manifest against the pinned public [Instavar Voice evaluation contract](https://github.com/instavar/instavar-voice-evaluation). New lifecycle and resume-evidence runs should use evaluator commit `29c38cfd86b889abc8b79df063c817dd8f684903` or a deliberately reviewed successor so POSIX stage timeouts clean the complete process group and schema 1.1 receipts bind live conditioning artifacts. This does not retroactively upgrade earlier run evidence.
 
 The lifecycle preserves invalid generations as explicit rows, then uses
 evaluator revision `8feadf7bbda75abe1c305c63e362c41b86451cda` to bind timing,

instavar-voice-capabilities.json

@@ -48,6 +48,11 @@
               "level": "repository_declared",
               "ref": "README.md#guarded-epoch-boundary-continuation",
               "claim": "New single-process torch_ddp runs can publish content-bound epoch continuation packages with optimizer, scheduler, scaler, RNG, progress, and CV monitor state."
+            },
+            {
+              "level": "repository_declared",
+              "ref": "reports/resume-evaluator-045-instrumentation-2026-08-14.md",
+              "claim": "Future checkpoints expose independent model, optimizer, scheduler, trainer, and RNG files for evaluator 0.45 while preserving legacy loading."
             }
           ],
           "boundary": "Historical checkpoint preservation does not validate the new resume contract or establish merged-runtime equivalence."
@@ -445,6 +450,7 @@
     "The corrected inference_instruct2 run produced valid WAVs but universal requested-text WER failure and frequent post-hoc exact instruction overlap; test any frontend or instruction rewrite under a new frozen plan.",
     "The early-stop helper passed a two-rank control smoke, but a real multi-rank production training run remains unverified.",
     "Guarded single-process epoch continuation has dependency-free contract coverage but no real interrupted GPU comparison.",
+    "Existing resume evidence predates evaluator 0.45 live-conditioning receipts; a fresh paired run is required for the stronger byte-exact claim tier.",
     "DeepSpeed, multi-rank, and multi-worker guarded continuation are rejected until rank-local and worker state plus collective publication are represented.",
     "Run a real PyTorch lifecycle package through managed backup and verified restore after completing model, dataset, speaker, and generated-output rights review.",
     "The 18-row split-boundary probe showed that the earlier request-seed association follows frontend request boundaries rather than text length alone. Test a per-utterance seed schedule and reset policy only after selecting a checkpoint that passes the content gate."

reports/resume-evaluator-045-instrumentation-2026-08-14.md

@@ -0,0 +1,62 @@
+# Evaluator 0.45 resume instrumentation
+
+Date: 2026-08-14, Asia/Singapore
+
+## Change
+
+Future guarded CosyVoice3 LoRA checkpoints write sidecar-bound
+`optimizer-state.pt` and `scheduler-state.pt` files in addition to the combined
+`runtime-state.pt` loader source. The five evaluator 0.45 roles now map as
+follows:
+
+| Evaluator role | CosyVoice checkpoint member |
+| --- | --- |
+| `model_state` | `adapter_model.safetensors` |
+| `optimizer_state` | `optimizer-state.pt` |
+| `scheduler_state` | `scheduler-state.pt` |
+| `trainer_state` | `training-state.json` |
+| `rng_state` | `runtime-state.pt` |
+
+The runtime file also contains optimizer, scheduler, AMP scaler, and RNG state
+because the existing loader consumes that combined structure. The decomposed
+optimizer and scheduler copies make the required file roles independently
+addressable without breaking the old loader. Older guarded checkpoints that
+contain only the combined file remain resumable under their original sidecars.
+
+`evaluator_lora_artifact_paths(...)` requires the new files, rechecks the live
+sidecar manifest, and rejects ambiguous adapter files or cross-role hardlinks.
+
+## OOD and compatibility controls
+
+Dependency-free tests cover:
+
+- the five-role mapping for a new checkpoint;
+- a legacy combined-only checkpoint that still resumes but cannot claim 0.45
+  readiness;
+- ambiguous adapter model files;
+- cross-role optimizer and scheduler hardlinks;
+- source-level confirmation that the trainer writes both decomposed files;
+- sidecar and continuation-state byte drift;
+- interruption during partial publication; and
+- unsupported DeepSpeed, multi-rank, and completed-target continuation.
+
+The public contract workflow pins evaluator revision
+`29c38cfd86b889abc8b79df063c817dd8f684903` and verifies its schema 1.1 receipt
+builder and comparison APIs.
+
+## Evidence boundary
+
+No model training or GPU test was run for this instrumentation change. It
+establishes repository behavior and dependency-free contract coverage only.
+Historical checkpoints and the selected epoch-12 adapter predate the new files
+and schema 1.1 live-conditioning receipts. They are not upgraded.
+
+A stronger comparison must preregister and fingerprint the Base artifact,
+dataset-lineage receipt, training controls, and initial state, then build
+independent uninterrupted and interrupted-resumed receipts before inspecting
+the outcome. Both runs must reach the same target update and the resumed run
+must bind an observed interruption before that target.
+
+Even a passing comparison establishes byte equality only for the declared
+files. It does not prove trainer semantics, hidden floating-point equivalence,
+adaptation benefit, perceptual quality, vLLM equivalence, or distributed resume.

tests/test_resume_contract.py

@@ -1,17 +1,21 @@
 from __future__ import annotations
 
 import json
+import os
 import tempfile
 import unittest
 from pathlib import Path
 
 from tools.cosyvoice_resume_contract import (
     LOCK_NAME,
+    OPTIMIZER_STATE_NAME,
     ResumeContractError,
+    SCHEDULER_STATE_NAME,
     acquire_output_lock,
     build_contract,
     checkpoint_children,
     epoch_checkpoint_name,
+    evaluator_lora_artifact_paths,
     prune_owned_checkpoints,
     publish_checkpoint,
     require_fresh_output,
@@ -72,6 +76,8 @@ def _adapter_saver(directory: Path) -> None:
     @staticmethod
     def _runtime_saver(path: Path) -> None:
         path.write_bytes(b"trusted-pickle-fixture")
+        (path.parent / OPTIMIZER_STATE_NAME).write_bytes(b"optimizer-fixture")
+        (path.parent / SCHEDULER_STATE_NAME).write_bytes(b"scheduler-fixture")
 
     def _checkpoint(
         self,
@@ -110,6 +116,97 @@ def test_exact_trusted_checkpoint_validates(self) -> None:
         self.assertEqual(state["completed_epoch"], 1)
         self.assertEqual(state["monitor_state"]["cv_no_improve_epochs"], 1)
 
+    def test_evaluator_lora_artifact_roles_are_independent(self) -> None:
+        checkpoint = self._checkpoint(1)
+        artifacts = evaluator_lora_artifact_paths(checkpoint)
+        self.assertEqual(
+            {role: path.name for role, path in artifacts.items()},
+            {
+                "model_state": "adapter_model.safetensors",
+                "optimizer_state": OPTIMIZER_STATE_NAME,
+                "scheduler_state": SCHEDULER_STATE_NAME,
+                "trainer_state": "training-state.json",
+                "rng_state": "runtime-state.pt",
+            },
+        )
+
+    def test_legacy_combined_runtime_state_remains_resumable(self) -> None:
+        checkpoint = publish_checkpoint(
+            output_dir=self.output,
+            completed_epoch=1,
+            completed_step=10,
+            contract=self.contract,
+            adapter_saver=self._adapter_saver,
+            runtime_state_saver=lambda path: path.write_bytes(b"legacy-combined-state"),
+            monitor_state={"best_cv_loss": 3.0},
+        )
+        selected, _ = validate_checkpoint(
+            checkpoint,
+            output_dir=self.output,
+            expected_contract=self.contract,
+            trust_resume_state=True,
+            world_size=1,
+            train_engine="torch_ddp",
+        )
+        self.assertEqual(selected, checkpoint)
+        with self.assertRaisesRegex(ResumeContractError, "omits decomposed state"):
+            evaluator_lora_artifact_paths(checkpoint)
+
+    def test_evaluator_mapping_rejects_ambiguous_model_and_hardlinks(self) -> None:
+        def ambiguous_adapter(directory: Path) -> None:
+            self._adapter_saver(directory)
+            (directory / "adapter_model.bin").write_bytes(b"second-adapter")
+
+        ambiguous = publish_checkpoint(
+            output_dir=self.output,
+            completed_epoch=1,
+            completed_step=10,
+            contract=self.contract,
+            adapter_saver=ambiguous_adapter,
+            runtime_state_saver=self._runtime_saver,
+            monitor_state={},
+        )
+        with self.assertRaisesRegex(ResumeContractError, "exactly one adapter"):
+            evaluator_lora_artifact_paths(ambiguous)
+
+        other_output = self.root / "hardlink-output"
+        other_output.mkdir()
+        hardlink_contract = build_contract(
+            output_dir=other_output,
+            base_checkpoint=self.base,
+            config_file=self.config,
+            qwen_pretrain=self.qwen,
+            data_files={"train": [self.train_list], "cross_validation": [self.cv_list]},
+            source_files=[self.source],
+            training_config={"max_epoch": 5, "train_engine": "torch_ddp"},
+            runtime={"python": "fixture", "world_size": 1},
+        )
+
+        def hardlinked_runtime(path: Path) -> None:
+            path.write_bytes(b"runtime")
+            optimizer = path.parent / OPTIMIZER_STATE_NAME
+            optimizer.write_bytes(b"shared-state")
+            os.link(optimizer, path.parent / SCHEDULER_STATE_NAME)
+
+        hardlinked = publish_checkpoint(
+            output_dir=other_output,
+            completed_epoch=1,
+            completed_step=10,
+            contract=hardlink_contract,
+            adapter_saver=self._adapter_saver,
+            runtime_state_saver=hardlinked_runtime,
+            monitor_state={},
+        )
+        with self.assertRaisesRegex(ResumeContractError, "must not share hardlinks"):
+            evaluator_lora_artifact_paths(hardlinked)
+
+    def test_trainer_writes_decomposed_state_before_publication(self) -> None:
+        source = (
+            Path(__file__).parents[1] / "tools" / "train_cosyvoice3_lora.py"
+        ).read_text(encoding="utf-8")
+        self.assertIn("path.parent / OPTIMIZER_STATE_NAME", source)
+        self.assertIn("path.parent / SCHEDULER_STATE_NAME", source)
+
     def test_resume_requires_explicit_trust(self) -> None:
         checkpoint = self._checkpoint(1)
         with self.assertRaisesRegex(ResumeContractError, "pickle-capable"):

tools/cosyvoice_resume_contract.py

@@ -18,6 +18,8 @@
 SIDECAR_NAME = "resume-contract.json"
 STATE_NAME = "training-state.json"
 RUNTIME_STATE_NAME = "runtime-state.pt"
+OPTIMIZER_STATE_NAME = "optimizer-state.pt"
+SCHEDULER_STATE_NAME = "scheduler-state.pt"
 LOCK_NAME = ".instavar-training.lock"
 _CHECKPOINT_RE = re.compile(r"^resume_epoch_(\d{6})$")
 
@@ -376,6 +378,65 @@ def _validate_checkpoint_files(
     return sidecar
 
 
+def evaluator_lora_artifact_paths(checkpoint_dir: str | Path) -> dict[str, Path]:
+    """Map a new guarded LoRA checkpoint to evaluator 0.45 state roles."""
+    raw = Path(checkpoint_dir).expanduser()
+    if raw.is_symlink():
+        raise ResumeContractError("Evaluator checkpoint must not be a symlink")
+    checkpoint = raw.resolve(strict=True)
+    if not checkpoint.is_dir() or not _CHECKPOINT_RE.fullmatch(checkpoint.name):
+        raise ResumeContractError("Evaluator checkpoint must be a guarded epoch directory")
+
+    sidecar_path = checkpoint / SIDECAR_NAME
+    if sidecar_path.is_symlink() or not sidecar_path.is_file():
+        raise ResumeContractError(f"Checkpoint has no safe {SIDECAR_NAME}")
+    try:
+        sidecar = json.loads(sidecar_path.read_text(encoding="utf-8"))
+    except json.JSONDecodeError as error:
+        raise ResumeContractError("Checkpoint sidecar is invalid JSON") from error
+    if sidecar.get("schema_version") != SCHEMA_VERSION:
+        raise ResumeContractError("Unsupported guarded checkpoint schema")
+    if sidecar.get("checkpoint_name") != checkpoint.name:
+        raise ResumeContractError("Checkpoint sidecar names a different directory")
+    manifest = sidecar.get("files")
+    if not isinstance(manifest, list) or manifest != _required_manifest(checkpoint):
+        raise ResumeContractError("Guarded checkpoint file identity drift detected")
+
+    names = {item["path"] for item in manifest}
+    required = {
+        "adapter_model.safetensors",
+        OPTIMIZER_STATE_NAME,
+        SCHEDULER_STATE_NAME,
+        STATE_NAME,
+        RUNTIME_STATE_NAME,
+    }
+    missing = sorted(required - names)
+    if missing:
+        raise ResumeContractError(
+            "Evaluator mapping omits decomposed state: " + ", ".join(missing)
+        )
+    model_candidates = sorted(
+        name
+        for name in names
+        if name in {"adapter_model.safetensors", "adapter_model.bin"}
+    )
+    if len(model_candidates) != 1:
+        raise ResumeContractError("Evaluator mapping needs exactly one adapter model state")
+
+    relative_by_role = {
+        "model_state": model_candidates[0],
+        "optimizer_state": OPTIMIZER_STATE_NAME,
+        "scheduler_state": SCHEDULER_STATE_NAME,
+        "trainer_state": STATE_NAME,
+        "rng_state": RUNTIME_STATE_NAME,
+    }
+    resolved = {role: checkpoint / name for role, name in relative_by_role.items()}
+    identities = [(path.stat().st_dev, path.stat().st_ino) for path in resolved.values()]
+    if len(identities) != len(set(identities)):
+        raise ResumeContractError("Evaluator artifact roles must not share hardlinks")
+    return resolved
+
+
 def prune_owned_checkpoints(
     output_dir: str | Path,
     *,

tools/train_cosyvoice3_lora.py

@@ -23,7 +23,9 @@
 
 from distributed_early_stop import synchronize_early_stop
 from cosyvoice_resume_contract import (
+    OPTIMIZER_STATE_NAME,
     ResumeContractError,
+    SCHEDULER_STATE_NAME,
     acquire_output_lock,
     build_contract,
     prune_owned_checkpoints,
@@ -201,9 +203,15 @@ def save_runtime_state(path: Path, optimizer, scheduler, scaler) -> None:
         "torch_rng": torch.get_rng_state(),
         "cuda_rng": torch.cuda.get_rng_state_all() if torch.cuda.is_available() else [],
     }
-    torch.save(state, path)
-    with path.open("rb") as handle:
-        os.fsync(handle.fileno())
+    outputs = {
+        path: state,
+        path.parent / OPTIMIZER_STATE_NAME: state["optimizer"],
+        path.parent / SCHEDULER_STATE_NAME: state["scheduler"],
+    }
+    for output, value in outputs.items():
+        torch.save(value, output)
+        with output.open("rb") as handle:
+            os.fsync(handle.fileno())
 
 
 def restore_runtime_state(path: Path, optimizer, scheduler, scaler) -> None: