[Node Operator Question] Node syncing but consistently ~2200 blocks behind

[Incede]

Are you running the most up to date node software?

• Yes

Did you check the documentation?

• Yes

Did you check for duplicate questions?

• Yes

Issue Description

Node stopped syncing around 2025-03-15 01 UTC at block 18343482.

Started syncing blocks again since around 2025-03-15 13:28:00 UTC but at a slower rate than the chain is producing blocks so not catching up. The graph shows the block height of the node vs the block height of head chain and after the restart it falls behind further but then catches up to about the same number of block behind.

Steps to Reproduce

Arguments used for op-node
--l2.jwt-secret=/home/op/jwtsecret --l2=ws://localhost:8551/ --rpc.addr=0.0.0.0 --metrics.enabled --metrics.addr=0.0.0.0 --metrics.port=7300 --l1="***" --l1.beacon="***" --rollup.config=/home/op/rollup.json --network=lisk-sepolia --rpc.port=9545 --p2p.bootnodes="enr:***" --p2p.static="***,***,***" --rollup.load-protocol-versions=true --verifier.l1-confs=4 --override.holocene=1732633200

And the arguments for op-geth
--ws --ws.addr=0.0.0.0 --ws.origins="*" --http --http.addr=0.0.0.0 --http.vhosts="*" --http.corsdomain="*" --authrpc.addr=localhost --authrpc.jwtsecret=/home/op/jwtsecret --authrpc.port=8551 --authrpc.vhosts="*" --datadir=/home/op/.datadir --metrics --metrics.addr=0.0.0.0 --metrics.port=6060 --op-network=lisk-sepolia --override.holocene=1732633200 --rollup.halt=major --rollup.superchain-upgrades --gcmode=archive --syncmode=full --op-network=lisk-sepolia --rollup.sequencerhttp=https://rpc.sepolia-api.lisk.com/ --http.api=eth,net,web3,debug --http.port=8545 --ws.port=8546 --snapshot=true --cache=4096 --maxpeers=0 --rollup.disabletxpoolgossip=true

Which component is affected?

• op-geth
• op-node
• Other

Expected vs. Actual Behavior

Node should catch up but doesn't

Environment Details

Linux

Protocol Information

op-node and op-geth were already upgraded to versions 1.12.0 and 1.101503.0.
network name : lisk-sepolia

Node Logs

In the logs, saw these two errors after the restart

Mar 18 09:07:43 juju-c860aa-0 op-node[1663296]: t=2025-03-18T09:07:43+0000 lvl=error msg="Cannot configure network and rollup-config at the same time.\nStartup will proceed to use the network-parameter and ignore the rollup config.\nConflicting configuration is deprecated, and will stop the op-node from starting in the future.\n" network=lisk-sepolia rollup_config=/home/op/rollup.json
...
Mar 18 09:07:46 juju-c860aa-0 op-node[1663296]: t=2025-03-18T09:07:46+0000 lvl=warn msg="Deriver system is resetting" err="reset: cannot continue derivation until Engine has been reset"

Apart from that, some reoccurring warnings

Mar 18 00:20:32 juju-c860aa-0 op-node[1612996]: t=2025-03-18T00:20:32+0000 lvl=warn msg="dropping future span batch" origin=0x66867d8bf2dc88720a756540be264ce5e90c9892027f0c7ba7f10a2aff23d3f9:7924465 epoch=0x4f0a3ca33ac7d60555bd29b8f0e4f6409c652023215dcaaf9f2c9988a10bf362:7920865 batch_type=SpanBatch batch_timestamp=1742253638 parent_check=0x7cd2a8a5b806185955f7717221e0f61ff991f28a origin_check=0x158090a210e3d20a204eb220888bfb9651985c48 start_epoch_number=7924164 end_epoch_number=7924166 block_count=10 txs=0 next_timestamp=1742213904
Mar 18 00:20:32 juju-c860aa-0 op-node[1612996]: t=2025-03-18T00:20:32+0000 lvl=warn msg="Dropping invalid span batch, flushing channel" origin=0x66867d8bf2dc88720a756540be264ce5e90c9892027f0c7ba7f10a2aff23d3f9:7924465 epoch=0x4f0a3ca33ac7d60555bd29b8f0e4f6409c652023215dcaaf9f2c9988a10bf362:7920865 batch_type=SpanBatch batch_timestamp=1742253638 parent_check=0x7cd2a8a5b806185955f7717221e0f61ff991f28a origin_check=0x158090a210e3d20a204eb220888bfb9651985c48 start_epoch_number=7924164 end_epoch_number=7924166 block_count=10 txs=0
...
Mar 18 04:32:36 juju-c860aa-0 op-node[1612996]: t=2025-03-18T04:32:36+0000 lvl=warn msg="L1 head signal indicates a possible L1 re-org" old_l1_head=0x1b4f383aa669713c7f9472e9d240f36406f66bc5da8a0771d793c544ec029665:7925727 new_l1_head_parent=0x52784af2eaa1f29ca9a9f3dc6fd8c122fc9de8b6989d660f8d398194e06c3b8b new_l1_head=0x2097db75400e569342651e86f9d592d3ac679aaa2b60c94505c8d9373a7a4161:7925729

Troubleshooting Attempts

Adding --syncmode=execution-layer as argument to op-node, the only noticeable difference was that the node quickly caught up to about being the same amount of blocks behind again after the restart

Tried to stop the node slightly longer and the same behavior can be seen.
And also when just looking at block height both nodes are syncing at the same speed as head chain but is just ~2200 blocks behind all the time.

Additional Information

No rate limits on L1 RPC used and default public sequencer is used.
Block height vs head chain : https://i.postimg.cc/W1gRkM3z/1.png
https://i.postimg.cc/pLR4kqX4/2.png
https://i.postimg.cc/T30SsgKw/3.png
op-node logs : https://sharetext.io/06d4ce0d

Feedback

No response

[jsvisa]

The first error-log in your op-node said you can't both set --rollup.config=/home/op/rollup.json --network=lisk-sepolia, that's not the main problem.

I think the second one maybe the main issue: lvl=warn msg="Deriver system is resetting" err="reset: cannot continue derivation until Engine has been reset"

Btw, want to find more useful logs in your log https://sharetext.io/06d4ce0d, but this file is too large for my browser to open it, could you please upload it as a github gist https://gist.github.com/ :)

[jsvisa]

Sorry for the late reply, the logs you provided seems ok, just wait for the op-node a while to let it catch up.

And for the 2200 blocks lagging, I think maybe you need to check for the op-node/op-geth's p2p nodes, maybe there's too little p2p nodes for the lisk-sepolia chain, could you please take a look at the p2p RPCs for the op-node, refer https://docs.optimism.io/operators/node-operators/json-rpc#opp2p

[Incede]

op-node has 3 peers but op-geth has 0. It gets blobs/blocks and is not stuck but just far behind all the time.

 Mar 18 04:34:24 juju-c860aa-0 op-node[1612996]: t=2025-03-18T04:34:24+0000 lvl=info msg="Inserted new L2 unsafe block" hash=0x3bffd335521e83062c643843dc7ac83283d21cb09f9168fd6d9352975eafd285 number=18458078 state_root=0x7e86717774d4a256cb4d8c1143ae78a6f1ae5a46d1db951d4f48545304cce953 timestamp=1742229148 parent=0xc3ab20eb560969ec617cc693b7dde64668b12d08c16b12e5e7b148d151ba3e22 prev_randao=0x6aa77d49943e6e116285f11af05b1f60b105a7e4ad1e9031866119483d4a7e16 fee_recipient=0x4200000000000000000000000000000000000011 txs=1 build_time=866.204µs insert_time=1.509ms total_time=2.376ms mgas=0.043815 mgasps=18.43996584303029
 Mar 18 04:34:24 juju-c860aa-0 op-node[1612996]: t=2025-03-18T04:34:24+0000 lvl=info msg="skipping unsafe payload, since it does not build onto the existing unsafe chain" safe=0xc3ab20eb560969ec617cc693b7dde64668b12d08c16b12e5e7b148d151ba3e22:18458077 unsafe=0x3bffd335521e83062c643843dc7ac83283d21cb09f9168fd6d9352975eafd285:18458078 unsafe_payload=0xf275e18a741334ec8d949bf81efe13e72150201290d5252b9f29daff04ea5253:18458079
 Mar 18 04:34:24 juju-c860aa-0 op-node[1612996]: t=2025-03-18T04:34:24+0000 lvl=info msg="Generating next batch" epoch=0xafe5f16e19465ad0e84690d37a2184351339a7b94a67e3524300964281bf14ba:7922133 timestamp=1742229150
 Mar 18 04:34:24 juju-c860aa-0 op-node[1612996]: t=2025-03-18T04:34:24+0000 lvl=info msg="generated attributes in payload queue" txs=1 timestamp=1742229150
 Mar 18 04:34:24 juju-c860aa-0 op-node[1612996]: t=2025-03-18T04:34:24+0000 lvl=info msg="Inserted new L2 unsafe block" hash=0x01b4e5b46bd89dfbe14880b23ca305a97becd231469089ac05c1d09b4edc4b15 number=18458079 state_root=0xe1ed288e86ffe3a6e3bf205d7606bc3535f0a60d6d176fabbcf59ed9228d23bc timestamp=1742229150 parent=0x3bffd335521e83062c643843dc7ac83283d21cb09f9168fd6d9352975eafd285 prev_randao=0x6aa77d49943e6e116285f11af05b1f60b105a7e4ad1e9031866119483d4a7e16 fee_recipient=0x4200000000000000000000000000000000000011 txs=1 build_time=1.421ms insert_time=1.906ms total_time=3.328ms mgas=0.043815 mgasps=13.165509520092163
 Mar 18 04:34:24 juju-c860aa-0 op-node[1612996]: t=2025-03-18T04:34:24+0000 lvl=info msg="skipping unsafe payload, since it does not build onto the existing unsafe chain" safe=0x3bffd335521e83062c643843dc7ac83283d21cb09f9168fd6d9352975eafd285:18458078 unsafe=0x01b4e5b46bd89dfbe14880b23ca305a97becd231469089ac05c1d09b4edc4b15:18458079 unsafe_payload=0xc6921d73488c3d60142098cf30a363a7a9e6965ad77a6d8eb109288a3c4532e4:18458080
 Mar 18 04:34:24 juju-c860aa-0 op-node[1612996]: t=2025-03-18T04:34:24+0000 lvl=info msg="Generating next batch" epoch=0xafe5f16e19465ad0e84690d37a2184351339a7b94a67e3524300964281bf14ba:7922133 timestamp=1742229152
 Mar 18 04:34:24 juju-c860aa-0 op-node[1612996]: t=2025-03-18T04:34:24+0000 lvl=info msg="generated attributes in payload queue" txs=1 timestamp=1742229152
 Mar 18 04:34:24 juju-c860aa-0 op-node[1612996]: t=2025-03-18T04:34:24+0000 lvl=info msg="Inserted new L2 unsafe block" hash=0x37d0f1f1c6ff1ea5f0b1a400dba0ce99a31e84f03d165669f84386d9bb5626c8 number=18458080 state_root=0x432a377531d22b5a8b3ff1e5ac71be0ebd79986644257e3e5e0bfa471bf35057 timestamp=1742229152 parent=0x01b4e5b46bd89dfbe14880b23ca305a97becd231469089ac05c1d09b4edc4b15 prev_randao=0x6aa77d49943e6e116285f11af05b1f60b105a7e4ad1e9031866119483d4a7e16 fee_recipient=0x4200000000000000000000000000000000000011 txs=1 build_time=968.236µs insert_time=1.764ms total_time=2.733ms mgas=0.043815 mgasps=16.030460619178598
 Mar 18 04:34:24 juju-c860aa-0 op-node[1612996]: t=2025-03-18T04:34:24+0000 lvl=info msg="skipping unsafe payload, since it does not build onto the existing unsafe chain" safe=0x01b4e5b46bd89dfbe14880b23ca305a97becd231469089ac05c1d09b4edc4b15:18458079 unsafe=0x37d0f1f1c6ff1ea5f0b1a400dba0ce99a31e84f03d165669f84386d9bb5626c8:18458080 unsafe_payload=0x2e1c8cac773812002170f8f2fe07b02c7fd3a5669ec8be8c20ae9587d31af075:18458081
 Mar 18 04:34:24 juju-c860aa-0 op-node[1612996]: t=2025-03-18T04:34:24+0000 lvl=info msg="Generating next batch" epoch=0xafe5f16e19465ad0e84690d37a2184351339a7b94a67e3524300964281bf14ba:7922133 timestamp=1742229154
 Mar 18 04:34:24 juju-c860aa-0 op-node[1612996]: t=2025-03-18T04:34:24+0000 lvl=info msg="generated attributes in payload queue" txs=1 timestamp=1742229154

[opfocus]

Please confirm whether your number of lagging blocks is 20,000+. If so, it may mean that your op-batcher once stopped submitting for more than 12 hours.

[Incede]

No it was much less ~2200 blocks.

[Incede]

Following integrity check and comparing the block hashes, the chain was on a different fork. The node just suddenly got into a different fork around 2025-03-15 00:57 UTC at around block 18343599

After rewinding the node to before block 18343482 when the node started to have issues, it was able to sync back to head.

The clients and configs haven't been changed b/w when the node started having issues and today when it was rewounded.
Do you know what could have caused the node into a different fork?