The execution payload envelope duty stops after BLOCK_ERROR_ALREADY_KNOWN, and it asks for the wrong block root

[ovidiu-ssv-labs]

Summary

We found two defects in the execution payload envelope duty. Each one alone stops Anchor from
publishing the envelope.

1. When the beacon node answers BLOCK_ERROR_ALREADY_KNOWN to the block publication, Anchor stops the
   full task. It then does not do the envelope duty for that slot.
2. When Anchor does the envelope duty, it asks the beacon node for the envelope of its own local
   candidate block, and not of the block that the committee decided.

Together, at most one Anchor operator joins each envelope consensus instance. In a committee where
two of the three operators in the quorum are Anchor operators, the envelope duty then never completes,
and the blocks are empty. See issue #1294, "A committee with two Anchor operators and one Go SSV operator
makes empty blocks".

Test environment

• Network: ssv-mini, a local Kurtosis testnet with 4 SSV operators. The quorum is 3 of 4.
• Operators 1 and 2 run Anchor, commit 52ba589 on branch epbs (Anchor/v1.3.1-52ba589).
• Operators 3 and 4 run the Go SSV node, commit 613dd52c9 on branch epbs-gloas.
• Consensus client: Lodestar v1.46.0/aa8303e. All 4 operators use the same beacon node.
• Execution client: go-ethereum, glamsterdam-devnet-8 image.
• Gloas fork epoch: 4 (slot 128). The Boole fork is not scheduled.

Note: all 4 operators use one beacon node. Thus every operator builds the same block and the same
envelope, and every operator sends the block to the same beacon node. Only the first operator gets a
success answer.

Defect 1: BLOCK_ERROR_ALREADY_KNOWN stops the full task

All operators of the committee send the same reconstructed block. The beacon node accepts the first
block and answers BLOCK_ERROR_ALREADY_KNOWN to the others. This answer is normal.

Anchor treats this answer as Irrecoverable:

11:52:33.134036 INFO  Publishing signed block  slot=205
11:52:33.147757 DEBUG Request to beacon node failed
                      error=Irrecoverable("... message: \"BLOCK_ERROR_ALREADY_KNOWN\" ...")
11:52:33.150819 ERROR Error whilst producing block   error="Some endpoints failed, num_failed: 2 ..."

After this error there is no Fetching execution payload envelope line for slot 205, and Anchor does
not join the envelope consensus instance of that slot.

The correlation is exact and opposite on the two Anchor nodes. One node stopped the task on slots 128,
134, 166, 175, 203 and 205, and did the envelope duty on slots 139, 143, 153, 202 and 217. The other
node did the opposite.

The Go SSV node treats BLOCK_ERROR_ALREADY_KNOWN as a success on the Gloas path
(beacon/goclient/gloas_proposer.go). In the same window the beacon node gave this answer 172 times,
and all Go SSV block proposals were a success.

What we expect

Treat BLOCK_ERROR_ALREADY_KNOWN as a success. Continue with the envelope duty.

Defect 2: the envelope request uses the local candidate root

Anchor asks for the envelope of the block root of its own candidate block:

anchor-node-0 11:38:21.233871 INFO Fetching execution payload envelope
                                   slot=134 beacon_block_root=0xfc43256b6aff...
canonical root of slot 134       = 0x9bcce65952...

The beacon node answers HTTP 404 for the root 0xfc43256b6aff....

8 of the 12 envelope requests used a root that the beacon node does not know. The root was correct
only when an Anchor operator was the leader of the block proposal consensus (slots 128, 153, 205 and
217).

Lodestar writes the root of the request into the envelope that it gives:

Produced execution payload envelope slot=134, blockRoot=0xfc43256b...
Produced execution payload envelope slot=134, blockRoot=0x9bcce659...   (two times)

All three envelopes have the same blockHash=0x9327.... Thus the cached envelope of Anchor has a
different root from the value that the committee decided. The content check then stops the
publication.

This is not an error of the root calculation. The partial signature of Anchor for the decided block
was a part of a valid signature in every block that the chain accepted, and also in the blocks where a
Go SSV operator was the leader.

What we expect

Ask for the envelope of the block root that the committee decided.

Defect 3 (small): the correct skip is also an error

Anchor has a correct message for the condition "another operator built the decided envelope":

INFO Cluster decided another operator's envelope, skipping publish (expected)   (4 times)

But the same condition also gives an error message:

ERROR Error whilst producing block
      error="Error signing execution payload envelope:
             SpecificError(EnvelopeNotBuiltLocally { local_root: 0x..., decided_root: 0x... })"

We counted 8 of these error messages. The blocks of those slots are on the chain. Please use only the
INFO message for this condition.

Effect

In a committee with 2 Anchor operators and 2 Go SSV operators, the envelope duty completed 11 of 11.
The Anchor operator that won the block publication race always joined the instance, so the quorum of 3
was possible.

In a committee with 2 Anchor operators and 1 Go SSV operator, the envelope duty completed 0 of 4, and
all 4 blocks are empty on the chain.

[ovidiu-ssv-labs]

Correction: the test build is one commit before PR #1286

We tested Anchor commit 52ba589. On branch epbs, this commit is the commit before the merge
commit e874c1e of PR #1286, "replace envelope QBFT with disseminate-and-sign (SIP-94 §6)".

e874c1e  feat: replace envelope QBFT with disseminate-and-sign (SIP-94 §6) (#1286)
52ba589  refactor(message_validator): delete the unused received_from plumbing (#1284)   <- our build

Thus this issue shows the old envelope design. PR #1286 deleted that design. We give the effect of
PR #1286 on each defect below. We did not test commit e874c1e. The statements below come from a
read of the code, and not from a test run.

Defect 1 stays. Its effect is now larger.

Anchor has no code for the BLOCK_ERROR_ALREADY_KNOWN answer. The behavior is in Lighthouse
e58ec88fe:

• In validator_client/validator_services/src/block_service.rs, the call
  sign_and_publish_block(...).await? stops the task before the envelope step.
• The function handle_block_post_error accepts only HTTP 202. Lodestar gives a status that is not
  202 with the message BLOCK_ERROR_ALREADY_KNOWN. Lighthouse then makes an Irrecoverable error.

PR #1286 does not change this code.

In the old design, each operator did the envelope duty alone. An operator that won the publication
race could join the instance. In the new design, only the builder operator disseminates the envelope.
If the beacon node answers BLOCK_ERROR_ALREADY_KNOWN to the builder operator, that operator stops
before the envelope step. It does not disseminate. The other operators then wait to the payload-due
deadline and get a dissemination timeout. The committee reveals no envelope.

This condition is not only a result of one shared beacon node. On a topology with more than one
beacon node, the beacon node of the builder operator can get the block from gossip before the builder
operator sends it. The answer is then the same.

PR #1286 removes defect 2

A non-builder operator no longer gets an envelope from the beacon node for this duty. The function
sign_block records a DecidedBlockContext with the field built_locally. Only the operator whose
block the committee decided disseminates its blinded envelope. The other operators wait for that
message, check it against the decided bindings, and sign it. The root of the builder operator is
equal to the decided root. Thus the wrong root does not happen.

But one log line stays. Lighthouse continues to call fetch_sign_and_publish_payload_envelope for
each operator with the local root of that operator. A non-builder operator thus continues to write
Fetching execution payload envelope beacon_block_root=<local root>, and continues to get HTTP 404.
This has no effect on the duty, because the task that sign_block starts makes the share. Remember
this when you read the logs of a new build.

Defect 3 stays, and you know about it

PR #1286 replaces EnvelopeNotBuiltLocally with EnvelopeNonBuilderDelegated and
EnvelopeExternalBuild. The shape is the same: an INFO message, and an Err to Lighthouse, which
writes Error whilst producing block. The description of PR #1286 gives this same statement, and it
gives sigp/lighthouse#9948 as the place for the fix. We agree. This part of the issue needs no change
in Anchor.

Effect on issue #1294

Issue #1294 now has a second cause. The Go SSV node on branch epbs-gloas, commit 613dd52c9, is
the commit in our test. It still uses the QBFT envelope design
(protocol/v2/ssv/runner/envelope.go). The description of PR #1286 gives this same statement. Thus a
committee with Anchor operators after e874c1e and Go SSV operators cannot complete the envelope
duty. This is a result of the two different designs, and not of defect 1. Empty blocks in a mixed
committee after that commit are expected. They stay expected until the Go SSV node ships the no-QBFT
design.

Next step

We will do the test again on commit e874c1e once the SSV operator is also updated to replace the envelope QBFT, and we will report the new data. We keep this issue open for defect 1.

[shane-moore]

Agree with the correction above: the remaining functional fix is the block-publication error that prevents the builder from reaching envelope dissemination.

At Lighthouse pin e58ec88fe, sign_and_publish_block(...).await? precedes the envelope step. One detail to preserve: handle_block_post_error already accepts HTTP 202 and other 2xx statuses. The uncovered case is the non-success response carrying BLOCK_ERROR_ALREADY_KNOWN. Verify the exact BN response and handle that case without treating unrelated invalid-block errors as success.

At current Anchor e874c1e, built_locally compares the local candidate with the decided block root. Non-builders start their signing task before returning the signed block to Lighthouse, so their shares no longer depend on the local envelope fetch. Builders still use Lighthouse's callback. Keep the functional fix focused on preserving that builder path.

There is a narrower scope than the reference to Lighthouse #9948 might suggest: its current diff skips envelopes for external-builder bids. It does not handle Anchor's EnvelopeNonBuilderDelegated sentinel or distinguish which SSV operator built a self-build block. The remaining non-builder error noise needs separate treatment.

Acceptance should demonstrate a reveal when the builder's block POST reports already-known, while genuine publication failures retain their error behavior. Source checked; no new runtime test performed.

[shane-moore]

Follow-up after #1297 merged: I traced this again on Anchor epbs at ff74ca08ed76950a7320f0bfd5a7f7f55c3cbd35, using its Lighthouse pin 1256bd99e1b6c5d1290241849354b7e27dbc0ed9. The publication error still gates the builder's envelope work. This was a source investigation: no #1293 implementation or new runtime reproduction has been done.

The current call path is:

1. Anchor constructs Lighthouse's BlockServiceBuilder. Lighthouse owns block production and HTTP publication.
2. Anchor's sign_block decides and threshold-signs the block, then starts the detached non-builder envelope task before returning the signed block.
3. After that callback returns, Lighthouse POSTs the signed block through its proposer/beacon-node fallback. Its error handler accepts HTTP 202 and other 2xx statuses, but a non-success response carrying BLOCK_ERROR_ALREADY_KNOWN becomes Irrecoverable. If fallback finds no success, sign_and_publish_block(...).await? exits before envelope GET/sign/POST.
4. The locally built envelope is still disseminated only inside Anchor's sign_execution_payload_envelope, which Lighthouse never reaches on that failure. Non-builders can be waiting correctly, but cannot sign without a dissemination.

Two distinctions matter when debugging:

• built_locally means local candidate root equals decided root, not unique QBFT leader identity. Several operators using the same candidate can satisfy it. If every eligible builder loses the publication race, none disseminates.
• feat: replace envelope QBFT with disseminate-and-sign (SIP-94 §6) #1286 removed the non-builder signing dependency on Lighthouse's local-root fetch. That fetch can still fail or produce skip-related error logs, but the detached signing task does not depend on it. Also, the race is at beacon-node publication/import: Anchor reconstructs the threshold-signed block from shares. Another operator's POST or beacon gossip can reach the BN first.

There is a concrete Lodestar lead at the reported revision aa8303e9a2007f80584d5eb6d914be0bb9012db8: publishBlockV2 already treats an already-known gossip-validation error as benign, but locally produced blocks skip that validation. Its later processBlock catch rethrows the error, while sanity checking throws already-known unless ignoreIfKnown is set. This is a candidate for a narrow duplicate-publication fix. Latest Lodestar and its existing issues/PRs still need checking before pursuing it.

The Go SSV workaround at the reported commit recognizes already-known response text. Useful precedent, but its broad substring match is not automatically the right contract for Lighthouse.

We considered these options:

Option	Benefits	Costs / unresolved work
Handle duplicate block publication in Lighthouse	Preserves VC ownership and future upstream improvements. The behavior can be generic, without SSV-specific callbacks.	Requires an upstream change and pin bump. Must distinguish a benign duplicate from genuine publication failures.
Correct Lodestar's duplicate-publication response	Preserves the existing Anchor/Lighthouse flow and addresses the concrete server-side path above.	Requires a Lodestar change or an existing upstream fix to be adopted. Does not establish behavior for other BNs.
Give Anchor an independent builder-envelope continuation	Can keep the envelope duty running after Lighthouse's block task exits, without changing Lighthouse. Reuses Anchor's existing signing/dissemination logic.	Anchor takes ownership of envelope HTTP orchestration, deadlines, retries and BN selection, and must prevent duplicate work with Lighthouse's callback.

My preference after discussing the maintenance tradeoff is to keep publication in Lighthouse. An Anchor-only continuation looks feasible as a medium PR touching client wiring, validator-store envelope orchestration and tests. We would still inherit Lighthouse's types, HTTP client and block-production improvements, but future VC envelope improvements would need manual porting. This includes BN affinity, retry/failover behavior, publication timing, builder routing and eventual stateless envelope handling. Lighthouse already has explicit TODOs for envelope BN affinity and proposer fallback.

If Lighthouse changes remain off the table, I would investigate the Lodestar path first. An Anchor-owned continuation remains a possible temporary workaround with a removal path. No implementation direction has been committed to.

For anyone pursuing the Anchor-only option, these constraints came out of the trace:

• There is no publication-result/error-classification hook in the pinned BlockServiceBuilder, and its fetch/sign/publish method is private. Catching the HTTP error inside Anchor's signing callback cannot work, because the POST happens after that callback returns.
• A possible design starts a builder task after threshold block signing, gated on a Gloas self-build decision with built_locally. It waits for the exact decided beacon block to be known, fetches the matching envelope, reuses existing binding checks/dissemination/signature collection, then publishes. A successful envelope GET alone is insufficient evidence of block import: Lodestar serves it from the production cache. Any block-visibility polling would be new Anchor behavior requiring validation.
• Bound waiting, HTTP attempts and signing by the payload-due deadline. Pass the existing BN/proposer clients from client wiring and retain the envelope-fetch client for publication. The current path is stateful: the publishing BN needs its cached blobs/proofs. Switching only the POST to the stateless API cannot supply missing blobs/proofs.
• Give the envelope operation one owner per operator/duty. Lighthouse publishes every successful signing callback result; its callback has no successful "skip" outcome. A delegated callback would therefore still need an error/sentinel or a different integration, leaving misleading Lighthouse error logs unresolved. Running both publishers adds races and duplicate submissions.
• Having Anchor also POST the block, or replacing the whole Gloas block service, expands ownership further. Restricting block POST to one operator does not eliminate beacon-gossip races.

Validation needs to cover the HTTP boundary. Existing builder_disseminates_signs_and_publishes and sign_block_then_matching_envelope_succeeds exercise callbacks and captured dissemination, not an actual block POST followed by envelope GET/POST. Add coverage through the real BlockService path with an already-known block response and require the envelope POST. Include genuine publication failure, absent block, binding mismatch, deadline expiry, and, for an independent worker, callback races and BN affinity. Then repeat the relevant Lodestar runtime scenario.

Keep adjacent work separate: Lighthouse #9948 is still open and its current diff only skips envelopes for external-builder bids; it does not address this duplicate POST or Anchor's non-builder sentinel. #1288 concerns dissemination admission. For mixed-client acceptance, use a Go revision with compatible disseminate-and-sign behavior: the original Go revision still used envelope QBFT, as explained in Ovi's correction, so that combination cannot validate the current envelope flow.

[shane-moore]

Closing: none of the three defects needs an Anchor change.

Defect 1 is beacon-node behaviour, and the tested Lodestar build was one commit short of the fix. Lodestar aa8303e rethrew ALREADY_KNOWN for locally produced blocks in the processBlock catch. Lodestar #9832, merged 2026-08-27, returns early when the block is already in fork choice and treats ALREADY_KNOWN on import as benign, so the publish succeeds and the envelope step runs. It shipped in Lodestar v1.47.0. Lighthouse beacon nodes already return success for a duplicate block, so this never triggered there. Given that, we are not going to special-case an already-known response in Lighthouse's block service.

Defect 2 was removed by #1286, per the correction above.

Defect 3 (the non-builder Error whilst producing block log) is tracked in sigp/lighthouse#9948.

For the rerun: Anchor epbs at or past 7caa8554 (includes #1297 and #1299), Lodestar v1.47.0 or later, and a Go SSV build with disseminate-and-sign so the envelope flow matches. Please reopen if the builder still skips the envelope step on that combination.