fix: import short-epoch ledger snapshots

Author: paolino

README.md

@@ -91,6 +91,9 @@ amaru-bootstrap-producer-<github-sha>.tar.gz
   producer race.
 - A CI-gated `amaru-run-bootstrap` proof that Amaru can open the
   produced ledger/chain stores and reach ledger startup.
+- A short-epoch Antithesis golden gate that samples slots `9`, `129`,
+  and `249` from a generated 120-slot-epoch ChainDB and imports them
+  through Amaru.
 
 The architecture, state machine, release boundary, and concurrency model
 are documented with diagrams in `docs/architecture.md`.
@@ -135,6 +138,8 @@ the Amaru bootstrap projection of the node-10.7.1 state:
 - Conway/Dijkstra pool state projected to the fields Amaru imports
 - Conway/Dijkstra account state projected to Amaru's legacy delegation
   wrapper
+- empty reward-update state projected as a completed zero reward update,
+  matching Amaru's `has_rewards=true` import path
 
 ## Inputs / outputs
 
@@ -160,6 +165,7 @@ running.
 ├── chain.<network>.db/                    # populated by amaru import-headers/import-nonces
 ├── ledger.<network>.db/                   # populated by amaru import-ledger-state
 ├── snapshots/<slot>.<hash>.cbor           # target plus two prior epoch snapshots
+├── snapshots/history.<slot>.<hash>.json   # testnet era history sidecars
 ├── nonces.json                            # tail rewritten to previous-epoch header hash
 └── headers/header.<slot>.<hash>.cbor      # headers needed by Amaru
 ```
@@ -168,6 +174,11 @@ The latest snapshot's `<slot>.<hash>` must have a matching
 `headers/header.<slot>.<hash>.cbor`; Amaru uses that exact header when
 aligning its chain store to the ledger tip at startup.
 
+For custom testnets, the producer also corrects each open-ended current
+era history sidecar to the `epochLength` from the node's Shelley genesis.
+This matters for short-epoch Antithesis networks because Amaru imports
+testnet snapshots using the sidecar history file next to each snapshot.
+
 ## Local verification
 
 ```bash
@@ -180,6 +191,9 @@ Phase 0 smoke verdict and accepts either `PASS` or the expected
 bootstrap-producer verifier. The pure producer-specific end-to-end check
 is `.#checks.x86_64-linux.bootstrap-producer-synthesized`; the startup
 proof is `.#checks.x86_64-linux.amaru-run-bootstrap`.
+The short-epoch Antithesis regression checks are
+`.#checks.x86_64-linux.antithesis-short-epoch-samples` and
+`.#checks.x86_64-linux.antithesis-short-epoch-golden`.
 
 These checks prove bundle production, Amaru import, and Amaru startup
 alignment. They are not a full mainnet ledger-content coverage suite for

amaru-bootstrap.cabal

@@ -40,6 +40,7 @@ library
     , cardano-ledger-shelley
     , cardano-binary
     , cardano-slotting
+    , cardano-strict-containers
     , cborg
     , containers
     , directory

docs/bootstrap-producer.md

@@ -54,13 +54,15 @@ signal for downstream Amaru services.
 4. Run `ledger-state-emitter` at the selected target slot and the two
    preceding epoch slots.
 5. Run `amaru convert-ledger-state` for all emitted ledger states.
-6. Run `header-extractor list-blocks` and `get-header` to collect the
+6. Correct the converted current-era history sidecars from the node
+   genesis `epochLength`.
+7. Run `header-extractor list-blocks` and `get-header` to collect the
    headers Amaru needs.
-7. Rewrite `nonces.json` so `tail` points at the previous-epoch header
+8. Rewrite `nonces.json` so `tail` points at the previous-epoch header
    hash.
-8. Run `amaru import-ledger-state`, `amaru import-headers`, and
+9. Run `amaru import-ledger-state`, `amaru import-headers`, and
    `amaru import-nonces`.
-9. Atomically rename the unique temp directory into the final bundle
+10. Atomically rename the unique temp directory into the final bundle
    path.
 
 The final layout is:
@@ -87,6 +89,14 @@ If the latest converted snapshot is
 `chain.<network>.db/`; otherwise `amaru run` fails during startup with
 `ledger tip header not found`.
 
+For custom testnets, `amaru import-ledger-state` reads
+`snapshots/history.<slot>.<hash>.json` next to each snapshot. Amaru's
+converter currently fills the open-ended current era with the network
+default epoch size, so the producer rewrites that sidecar to the
+`epochLength` from the mounted Shelley genesis before import. Without
+that correction a 120-slot Antithesis testnet imports slot 9, then fails
+at slot 129 because the sidecar still maps slot 129 to epoch 0.
+
 ## Node-Release Target
 
 This implementation targets `cardano-node 10.7.1`. The repository pins
@@ -154,6 +164,9 @@ node-10.7.1 ledger state:
 - Conway/Dijkstra account state is projected into Amaru's legacy
   delegation-state wrapper while preserving rewards, deposits,
   stake-pool delegation, and DRep delegation.
+- Empty reward-update state is projected as a completed zero reward
+  update because Amaru's import command decodes snapshots with
+  `has_rewards=true`.
 
 The detailed contract is in
 `specs/003-amaru-bootstrap-producer/research.md#r-011`.
@@ -200,7 +213,9 @@ shape a long-running public network can contain.
 `antithesis-short-epoch-samples` generates a deterministic short-epoch
 ChainDB corpus from the pinned node 10.7.1 tooling, emits the observed
 early bootstrap slots `9`, `129`, and `249`, and converts them through
-`amaru convert-ledger-state`. The source ChainDB is generated during the
+`amaru convert-ledger-state`. The check also rewrites the current-era
+history sidecars to the generated Shelley genesis `epochLength`, matching
+the production producer path. The source ChainDB is generated during the
 Nix build; the repository does not commit bulky database artifacts.
 
 `antithesis-short-epoch-golden` imports those converted snapshots into

docs/what-amaru-needs.md

@@ -11,6 +11,7 @@ Reverse-engineered from
 ├── chain.<network>.db/                    # amaru's chain store, prepopulated
 ├── ledger.<network>.db/                   # amaru's ledger store, prepopulated
 ├── snapshots/<slot>.<hash>.cbor           # target plus two prior epoch boundaries
+├── snapshots/history.<slot>.<hash>.json   # testnet era history sidecar
 ├── nonces.json                            # tail points to previous-epoch header hash
 └── headers/header.<slot>.<hash>.cbor      # includes the latest snapshot header
 ```
@@ -94,3 +95,9 @@ dependency set and emits the Amaru bootstrap projection documented in
 it three times: `target_slot`, `target_slot - epochLength`, and
 `target_slot - 2 * epochLength`. `amaru convert-ledger-state` still owns
 the final snapshot slicing, history JSON, and nonce JSON formats.
+
+For custom testnets, the producer corrects the converted
+`history.<slot>.<hash>.json` files before import: the open-ended current
+era's `epoch_size_slots` is set to the mounted Shelley genesis
+`epochLength`. This keeps short-epoch networks consistent with the
+ledger snapshot epoch number that Amaru checks during import.

justfile

@@ -40,6 +40,8 @@ build-gate:
         .#checks.x86_64-linux.bootstrap-producer-bats \
         .#checks.x86_64-linux.bootstrap-producer-synthesized \
         .#checks.x86_64-linux.amaru-run-bootstrap \
+        .#checks.x86_64-linux.antithesis-short-epoch-samples \
+        .#checks.x86_64-linux.antithesis-short-epoch-golden \
         .#checks.x86_64-linux.bootstrap-producer-image
 
 # Run the unit-style bats checks.

lib/LedgerStateEmitter.hs

@@ -39,6 +39,9 @@ adjusted where Amaru's importer still expects older ledger shapes:
   and DRep delegation are preserved; historical pointer indexes and the
   intermediate deposits accumulator are omitted because Amaru skips them
   during bootstrap.
+* Empty reward-update state is projected as a completed zero reward
+  update. Amaru's import command runs in @has_rewards=true@ mode and
+  unconditionally decodes a @Complete RewardUpdate@ at this position.
 -}
 module LedgerStateEmitter (
     emitLedgerSnapshot,
@@ -58,6 +61,7 @@ import Cardano.Ledger.Conway.State qualified as Conway
 import Cardano.Ledger.Core qualified as Core
 import Cardano.Ledger.Dijkstra.State ()
 import Cardano.Ledger.Shelley.LedgerState qualified as SL
+import Cardano.Ledger.Shelley.RewardUpdate qualified as ShelleyReward
 import Cardano.Ledger.State qualified as Ledger
 import Cardano.Slotting.Slot (SlotNo)
 import Cardano.Tools.DBAnalyser.Block.Cardano (
@@ -81,11 +85,13 @@ import Control.ResourceRegistry (
 import Data.ByteString.Lazy qualified as LBS
 import Data.Functor.Contravariant ((>$<))
 import Data.Map.Strict qualified as Map
+import Data.Maybe.Strict (StrictMaybe (..))
 import Data.Proxy (Proxy (Proxy))
 import Data.SOP.BasicFunctors (K (K))
 import Data.SOP.Functors (Flip (unFlip))
 import Data.SOP.Strict (NP (Nil, (:*)), fn, type (-.->))
 import HeaderExtractor (NodeConfig (NodeConfig))
+import Lens.Micro ((^.))
 import Ouroboros.Consensus.Block (
     GetHeader,
     blockNo,
@@ -190,7 +196,6 @@ import Ouroboros.Consensus.Util.CBOR (
  )
 import Ouroboros.Consensus.Util.Versioned (encodeVersion)
 import Ouroboros.Network.Block (genesisPoint)
-import Lens.Micro ((^.))
 import System.Directory (createDirectoryIfMissing)
 import System.FilePath (takeDirectory)
 
@@ -499,7 +504,8 @@ encodeShelleyLedgerStateCanonicalWith ::
     (SL.NewEpochState era -> CBOR.Encoding) ->
     LedgerState (ShelleyBlock proto era) EmptyMK ->
     CBOR.Encoding
-encodeShelleyLedgerStateCanonicalWith encodeNewEpochState
+encodeShelleyLedgerStateCanonicalWith
+    encodeNewEpochState
     ShelleyLedgerState
         { shelleyLedgerTip
         , shelleyLedgerState
@@ -562,18 +568,29 @@ encodeNewEpochStateCanonicalLedgerWith encodeCertState (SL.NewEpochState e bp bc
         , encCBOR bp
         , encCBOR bc
         , encodeEpochStateCanonicalWith encodeCertState es
-        , encCBOR ru
+        , encodeRewardUpdateAmaru ru
         , encCBOR pd
         , encCBOR av
         ]
 
+encodeRewardUpdateAmaru ::
+    StrictMaybe ShelleyReward.PulsingRewUpdate ->
+    LedgerEncoding
+encodeRewardUpdateAmaru =
+    encCBOR . \case
+        SNothing ->
+            SJust $
+                ShelleyReward.Complete ShelleyReward.emptyRewardUpdate
+        SJust ru -> SJust ru
+
 encodeEpochStateCanonicalWith ::
     forall era.
     (CanonicalShelleyLedgerCbor era) =>
     (SL.CertState era -> LedgerEncoding) ->
     SL.EpochState era ->
     LedgerEncoding
-encodeEpochStateCanonicalWith encodeCertState
+encodeEpochStateCanonicalWith
+    encodeCertState
     SL.EpochState
         { SL.esChainAccountState = esChainAccountState
         , SL.esLState = esLState
@@ -594,7 +611,8 @@ encodeLedgerStateCanonicalWith ::
     (SL.CertState era -> LedgerEncoding) ->
     SL.LedgerState era ->
     LedgerEncoding
-encodeLedgerStateCanonicalWith encodeCertState
+encodeLedgerStateCanonicalWith
+    encodeCertState
     SL.LedgerState
         { SL.lsUTxOState = lsUTxOState
         , SL.lsCertState = lsCertState

nix/checks.nix

@@ -204,6 +204,7 @@ let
           pkgs.bash
           pkgs.coreutils
           pkgs.findutils
+          pkgs.jq
           amaruPkg
           headerExtractorPkgs.ledger-state-emitter
         ];
@@ -229,6 +230,16 @@ let
           --target-dir "$out/snapshots"
       done
 
+      epoch_length=$(jq -r '.epochLength' \
+        "$TMPDIR/config/shelley-genesis.json")
+      for history in "$out"/snapshots/history.*.json; do
+        tmp="$history.tmp"
+        jq --argjson epochLength "$epoch_length" \
+          '(.eras[] | select(.end == null) | .params.epoch_size_slots) = $epochLength' \
+          "$history" >"$tmp"
+        mv "$tmp" "$history"
+      done
+
       cbor_count=$(find "$out/snapshots" -maxdepth 1 \
         -name '*.cbor' | wc -l)
       if [ "$cbor_count" -ne 3 ]; then
@@ -354,6 +365,8 @@ in
       BOOTSTRAP_PRODUCER_CHAIN_DB=$TMPDIR/chain-db \
         bats --tap tests/test-bootstrap-producer-concurrent.bats
 
+      bats --tap tests/test-bootstrap-producer-history.bats
+
       mkdir -p $out
     '';
 

scripts/bootstrap-producer.sh

@@ -102,6 +102,32 @@ tail_phase_log() {
     printf -- '--- end %s ---\n' "${f}" >&2
 }
 
+patch_converted_era_history() {
+    local history tmp count=0
+
+    for history in "${UNIQUE_TMP}/snapshots"/history.*.json; do
+        [[ -e "${history}" ]] || continue
+        count=$(( count + 1 ))
+        tmp="${history}.tmp"
+        if ! jq --argjson epochLength "${EPOCH_LENGTH}" \
+            '(.eras[] | select(.end == null) | .params.epoch_size_slots) = $epochLength' \
+            "${history}" >"${tmp}" 2>"${BUNDLE_DIR}/.logs/history.stderr"
+        then
+            printf 'era-history patch failed for %s; see %s\n' \
+                   "${history}" "${BUNDLE_DIR}/.logs/history.stderr" >&2
+            rm -f "${tmp}"
+            exit 6
+        fi
+        mv "${tmp}" "${history}"
+    done
+
+    if (( count == 0 )); then
+        printf 'amaru convert-ledger-state produced no history.*.json files in %s\n' \
+               "${UNIQUE_TMP}/snapshots" >&2
+        exit 6
+    fi
+}
+
 # ─── 8-step state diagram (functions stubbed, T018+T019 fill them) ─
 
 # Step 1: pre-flight (wait + validate + era-readiness predicate).
@@ -342,6 +368,9 @@ phase_emit() {
 # Writes <slot>.<hash>.cbor + nonces.<slot>.<hash>.json + history.<slot>.<hash>.json
 # into <staging>/snapshots/. amaru's import-ledger-state requires the
 # era-history file to live alongside the snapshot for testnet variants.
+# Amaru's converter currently fills the open-ended current era with the
+# network default epoch size; for custom short-epoch testnets the
+# producer corrects that sidecar from the node genesis before import.
 phase_convert() {
     local snapshot slot rc
     for snapshot in "${LEGACY_SNAPSHOT_FILES[@]}"; do
@@ -361,6 +390,8 @@ phase_convert() {
         fi
     done
 
+    patch_converted_era_history
+
     local cbor_count
     cbor_count=$(find "${UNIQUE_TMP}/snapshots" -maxdepth 1 \
         -name '*.cbor' 2>/dev/null | wc -l)

specs/004-short-epoch-golden-states/plan.md

@@ -11,15 +11,19 @@
 - Created issue #29 and moved it to WIP.
 - Reproduced the local import failure with generated short-epoch states:
   `unexpected type map at position 2: expected u32`.
+- Fixed the reward-update projection mismatch by emitting a completed
+  zero reward update when the node ledger state carries `SNothing`.
+- Fixed the short-epoch era-history mismatch by rewriting converted
+  current-era history sidecars from the node genesis `epochLength`.
+- Local `just build-gate` passes with the short-epoch checks included.
 
 **Current**:
 
-- Add generated sample and golden checks to Nix/CI.
+- Push the passing PR and let GitHub CI re-run the same Build Gate.
 
 **Blockers**:
 
-- The golden check is expected to fail until the Haskell projection or
-  Amaru importer contract is fixed.
+- None known locally.
 
 ## Technical Context
 
@@ -39,6 +43,11 @@ The repository should not grow committed ChainDB artifacts.
   chain is dense enough to expose immutable blocks in a small CI budget.
 - Sample slots `9`, `129`, and `249`, matching the observed early
   bootstrap emission points.
+- Project empty reward-update state as `Complete emptyRewardUpdate`
+  because Amaru's CLI imports snapshots with `has_rewards=true`.
+- Correct the converted open-ended current era history sidecar to the
+  Shelley genesis `epochLength`; Amaru's converter currently uses the
+  network default there, which is wrong for 120-slot custom testnets.
 - Split conversion and import:
   - `antithesis-short-epoch-samples` proves sample generation and
     conversion.
@@ -52,5 +61,5 @@ nix build .#checks.x86_64-linux.antithesis-short-epoch-samples
 nix build .#checks.x86_64-linux.antithesis-short-epoch-golden
 ```
 
-Before the fix, the second command is expected to fail at
-`amaru import-ledger-state` with the current CBOR/import mismatch.
+After the fix, both commands pass locally. `just build-gate` also passes
+with both short-epoch checks included in the gate.

specs/004-short-epoch-golden-states/spec.md

@@ -35,6 +35,9 @@ fail before the projection/import mismatch is fixed and pass afterwards.
   immutable blocks for `header-extractor` and `ledger-state-emitter`.
 - A check that only proves conversion is insufficient; the import step is
   the failing boundary.
+- Converted testnet history sidecars must agree with the node genesis
+  epoch length; otherwise Amaru maps slot 129 to epoch 0 while the
+  ledger snapshot declares epoch 1.
 - The corpus must remain pinned to the node 10.7.1 dependency set.
 
 ## Requirements
@@ -51,13 +54,16 @@ fail before the projection/import mismatch is fixed and pass afterwards.
   `amaru import-ledger-state`.
 - **FR-005**: The repository MUST document the generated corpus profile
   and explain why the database is not committed.
+- **FR-006**: The converted current-era history sidecars MUST use the
+  Shelley genesis `epochLength` before Amaru import on custom testnets.
 
 ### Key Entities
 
 - **Short-epoch ChainDB corpus**: Generated Cardano ChainDB configured to
   exercise the early Conway-from-genesis bootstrap state family.
 - **Golden snapshots**: Converted Amaru snapshots derived from the
-  sampled ledger states.
+  sampled ledger states, with history sidecars corrected to the sampled
+  genesis epoch length.
 - **Import gate**: CI check that requires Amaru to consume the snapshots.
 
 ## Success Criteria