A committee with two Anchor operators and one Go SSV operator makes empty blocks

[ovidiu-ssv-labs]

Summary

We removed one Go SSV operator from a mixed committee. This left two Anchor operators and one Go SSV
operator, which is exactly the quorum of 3 of 4.

In this configuration the committee cannot complete the execution payload envelope duty or the
aggregator duty. The proposer duty still completes, so the blocks are on the chain. But the payload is
never revealed. All 4 blocks that the committee proposed are empty.

This issue is a report of the effect. The two causes have their own issues:

• The start time of the aggregator duty and the sync committee contribution duty. See issue The aggregator duty and the sync committee contribution duty do not use the new Gloas start time #1292, "The aggregator duty and the sync committee contribution duty do not use the new Gloas start
  time".
• The two defects of the execution payload envelope duty. See issue The execution payload envelope duty stops after BLOCK_ERROR_ALREADY_KNOWN, and it asks for the wrong block root #1293, "The execution payload envelope
  duty stops after BLOCK_ERROR_ALREADY_KNOWN, and it asks for the wrong block root".

Test environment

• Network: ssv-mini, a local Kurtosis testnet with 4 SSV operators. The quorum is 3 of 4.
• Operators 1 and 2 run Anchor, commit 52ba589 on branch epbs (Anchor/v1.3.1-52ba589).
• Operators 3 and 4 run the Go SSV node, commit 613dd52c9 on branch epbs-gloas.
• Consensus client: Lodestar v1.46.0/aa8303e. All 4 operators use the same beacon node.
• Gloas fork epoch: 4 (slot 128). The Boole fork is not scheduled. 10 validators.

The test

We test what happens when one operator does not upgrade before the fork.

1. Let the network run after the Gloas fork. All 4 operators use a Gloas image.
2. Change the image of one operator to an old image that has no Gloas support.
3. Wait more than 2 epochs. Look at the duties of the 3 other operators.
4. Change the image back. Check that the operator joins the committee again.

We did this two times: one time with an Anchor operator as the old node, and one time with a Go SSV
operator as the old node. The old Go SSV image was v2.4.3. The old Anchor image was v1.3.1
without Gloas support.

The result of the two directions is very different. This issue is about the second direction, where
the old node is the Go SSV operator and two Anchor operators stay in the committee.

Result: old node is the Go SSV operator (2 Anchor + 1 Go SSV)

Window: 48 slots. Counts from the Go SSV operator that stayed in the committee.

Duty	Started	Completed
attester and sync committee message	48	48
proposer	4	4
proposer preferences	1	1
aggregator	18	0
execution payload envelope	4	0
sync committee contribution	480	366. 114 never completed

The committee submitted 0 aggregates and 0 envelopes.

Anchor logs in the same window, on each Anchor node:

• Signing aggregate timed out: 18. This is one for each of the 18 aggregator duties.
• Signing sync committee contribution timed out: 136.
• Envelope consensus timed out slot: 2 on one node, 1 on the other node.
• Successfully published sync contributions: 62. In the other direction of the test it was 198.

Go SSV logs for the aggregator duty: 18 x got pre-consensus quorum, then 18 x
starting QBFT instance, then 171 x round timed out, then 18 x
duty did not complete before slot end (likely stuck). 0 instances decided.

Go SSV logs for the execution payload envelope duty: 4 x built execution payload envelope, 4 x
starting QBFT instance, 42 x round timed out, 36 x ignoring invalid message, 4 x
duty did not complete before slot end.

The blocks are empty

In Gloas the block and the payload are separate. The proposer duty publishes the block with a signed
bid. The execution payload envelope duty then reveals the payload.

The proposer duty completed 4 of 4. The envelope duty completed 0 of 4. Thus the payload was never
revealed.

We read the payload attestation data from the next block of each proposed slot:

Configuration	Proposed slots	payload_present
1 Anchor + 2 Go SSV	802, 814, 834, 836, 841, 854, 863	true on 7 of 7
2 Anchor + 1 Go SSV	940, 954, 955, 958	false on 4 of 4

All 11 blocks are canonical and have version: gloas.

The validators get the reward for the block proposal. But the slots have no transactions.

The number of Anchor operators is the variable

We measured the aggregator duty in three configurations of the same committee, on the same day, with
the same images:

Operators in the quorum	Aggregator duties completed
2 Go SSV + 1 Anchor	20 of 20
2 Go SSV + 2 Anchor	7 of 7
1 Go SSV + 2 Anchor	0 of 18

The third row is the configuration of this issue.

Our analysis

For the aggregator duty and the sync committee contribution duty, the cause is the start time. Anchor
starts at two thirds of the slot, and the Go SSV node starts at one half. With two Go SSV operators
in the quorum, the two Go SSV operators keep the instance alive until Anchor joins. With only one Go
SSV operator, and with two late operators that then stop their own signature step
(Signing aggregate timed out on all 18 duties), the instance does not decide before the end of the
slot.

For the execution payload envelope duty, the cause is different and more direct: at most one Anchor
operator joins each envelope instance, because the other Anchor operator stopped its task earlier.
This leaves 2 signatures against a quorum of 3.

We have the symptoms from both sides and the correlation with the configuration. We do not have a
trace from inside the Anchor signature step. Please confirm the mechanism of Signing aggregate timed out.

What we expect

A committee with a quorum of two Anchor operators and one Go SSV operator must complete the aggregator
duty, the sync committee contribution duty and the execution payload envelope duty.

[shane-moore]

This is useful as the mixed-client acceptance issue for #1292 and #1293. Keep the topology explicit: a four-member committee with quorum three, with one operator unavailable, including both 2 Anchor + 1 Go SSV and 1 Anchor + 2 Go SSV survivor sets.

Two points for the next investigation:

• Pin compatible envelope implementations before rerunning. The tested Go SSV 613dd52c9 runner uses envelope QBFT. Anchor #1286, merged as e874c1e, replaces it with dissemination plus signing. Updating only Anchor changes the protocol under test. See Ovi's correction.
• Signing aggregate timed out does not establish that post-consensus signature collection timed out. At current Anchor, the generic signing wrapper emits that text for SpecificError::Timeout, and an aggregate QBFT timeout maps to that same error before signature collection begins. Check decision completion and partial-signature collection separately. The causal explanation for all 18 misses still needs that trace.

For acceptance, retain Gloas-on/Boole-off, exact image SHAs, both survivor sets, and operator recovery. Record aggregate/contribution decisions and publications, plus envelope reveals tied to the canonical proposed block roots. A successful block POST alone does not demonstrate a payload reveal.

These are code references and suggested evidence boundaries; I have not reproduced the reported losses.