[Node Operator Question] Under what conditions does the sequencer log "Sequencer paused until new events"?

[0xNarumi]

Are you running the most up to date node software?

• Yes

Did you check the documentation?

• Yes

Did you check for duplicate questions?

• Yes

Issue Description

I'm trying to understand under what conditions the sequencer logs Sequencer paused until new events.

In my case, the L2 chain was working fine and then suddenly stopped producing blocks with out any warning sign.

Here’s a op-node logs:

2025-05-29 08:44:50.211 | 2025-05-28T23:44:50.086729609Z stdout F t=2025-05-28T23:44:50+0000 lvl=info msg="Scheduled sequencer action" delta=-27.212µs |  
-- | -- | --
  |   | 2025-05-29 08:44:50.211 | 2025-05-28T23:44:50.086734557Z stdout F t=2025-05-28T23:44:50+0000 lvl=info msg="Started sequencing new block" parent=0xe5b06e85fa9010393d8c3b2a94647e1bf1aa67b2ae5c48d4e49433db39ef8ce8:13387130 l1Origin=0xdbda85dee62dd85cd7d314fb171bb8652563e4767dacbb31ada84583dff5f13d:8428128 |  
  |   | 2025-05-29 08:44:50.462 | 2025-05-28T23:44:50.401458223Z stdout F t=2025-05-28T23:44:50+0000 lvl=info msg="Scheduled sequencer action" delta=548.590757ms |  
  |   | 2025-05-29 08:44:51.212 | 2025-05-28T23:44:51.089304842Z stdout F t=2025-05-28T23:44:51+0000 lvl=info msg="Sequencer sealed block" payloadID=0x03fa25a3a9f80bed block=0xf3bf191642f1146b3ecdd72b880b98ed30871236c0e1ae2bc3519c6a76008401:13387131 parent=0xe5b06e85fa9010393d8c3b2a94647e1bf1aa67b2ae5c48d4e49433db39ef8ce8:13387130 txs=105 time=1748475891 |  
  |   | 2025-05-29 08:44:51.212 | 2025-05-28T23:44:51.089331661Z stdout F t=2025-05-28T23:44:51+0000 lvl=info msg="Sequencer inserted block" block=0xf3bf191642f1146b3ecdd72b880b98ed30871236c0e1ae2bc3519c6a76008401:13387131 parent=0xe5b06e85fa9010393d8c3b2a94647e1bf1aa67b2ae5c48d4e49433db39ef8ce8:13387130 |  
  |   | 2025-05-29 08:44:51.212 | 2025-05-28T23:44:51.089340495Z stdout F t=2025-05-28T23:44:51+0000 lvl=info msg="Publishing signed execution payload on p2p" id=0xf3bf191642f1146b3ecdd72b880b98ed30871236c0e1ae2bc3519c6a76008401:13387131 |  
  |   | 2025-05-29 08:44:51.212 | 2025-05-28T23:44:51.097693279Z stdout F t=2025-05-28T23:44:51+0000 lvl=info msg="Inserted new L2 unsafe block" hash=0xf3bf191642f1146b3ecdd72b880b98ed30871236c0e1ae2bc3519c6a76008401 number=13387131 state_root=0x4a80ea3846216bdea3851ac1d6a3ee1188a6e648b29666bed10ee2fb53f7514a timestamp=1748475891 parent=0xe5b06e85fa9010393d8c3b2a94647e1bf1aa67b2ae5c48d4e49433db39ef8ce8 prev_randao=0x26dc26d2b58c4cbf0e9b719eca40ce18546c1d3b00f1261d61c32373dd9ff2cf fee_recipient=0x4200000000000000000000000000000000000011 txs=105 build_time=-2562047h47m16.854s insert_time=2562047h47m16.854s total_time=1.011s mgas=6.663954 mgasps=6.590399343772081 |  
  |   | 2025-05-29 08:44:51.212 | 2025-05-28T23:44:51.097725219Z stdout F t=2025-05-28T23:44:51+0000 lvl=info msg="Scheduled sequencer action" delta=-21.574µs |  
  |   | 2025-05-29 08:44:51.212 | 2025-05-28T23:44:51.097784083Z stdout F t=2025-05-28T23:44:51+0000 lvl=info msg="Started sequencing new block" parent=0xf3bf191642f1146b3ecdd72b880b98ed30871236c0e1ae2bc3519c6a76008401:13387131 l1Origin=0xdbda85dee62dd85cd7d314fb171bb8652563e4767dacbb31ada84583dff5f13d:8428128 |  
  |   | 2025-05-29 08:44:51.463 | 2025-05-28T23:44:51.282008472Z stdout F t=2025-05-28T23:44:51+0000 lvl=info msg="Scheduled sequencer action" delta=668.043423ms |  
  |   | 2025-05-29 08:44:52.214 | 2025-05-28T23:44:52.038002853Z stdout F t=2025-05-28T23:44:52+0000 lvl=info msg="Sequencer sealed block" payloadID=0x033f1ae98873c2ec block=0x3d55356dfd639d4dbf7ec48063b728cedb48a3986cd4ed30841e40c490c0949d:13387132 parent=0xf3bf191642f1146b3ecdd72b880b98ed30871236c0e1ae2bc3519c6a76008401:13387131 txs=109 time=1748475892 |  
  |   | 2025-05-29 08:44:52.215 | 2025-05-28T23:44:52.038042594Z stdout F t=2025-05-28T23:44:52+0000 lvl=info msg="Sequencer inserted block" block=0x3d55356dfd639d4dbf7ec48063b728cedb48a3986cd4ed30841e40c490c0949d:13387132 parent=0xf3bf191642f1146b3ecdd72b880b98ed30871236c0e1ae2bc3519c6a76008401:13387131 |  
  |   | 2025-05-29 08:44:52.215 | 2025-05-28T23:44:52.03806633Z stdout F t=2025-05-28T23:44:52+0000 lvl=info msg="Publishing signed execution payload on p2p" id=0x3d55356dfd639d4dbf7ec48063b728cedb48a3986cd4ed30841e40c490c0949d:13387132 |  
  |   | 2025-05-29 08:44:52.216 | 2025-05-28T23:44:52.209980703Z stdout F t=2025-05-28T23:44:52+0000 lvl=info msg="Inserted new L2 unsafe block" hash=0x3d55356dfd639d4dbf7ec48063b728cedb48a3986cd4ed30841e40c490c0949d number=13387132 state_root=0x0b15f41a9ec2c29fb01dcaa02fa5f44791ef7e9baff6516e158dadbf043e2835 timestamp=1748475892 parent=0xf3bf191642f1146b3ecdd72b880b98ed30871236c0e1ae2bc3519c6a76008401 prev_randao=0x26dc26d2b58c4cbf0e9b719eca40ce18546c1d3b00f1261d61c32373dd9ff2cf fee_recipient=0x4200000000000000000000000000000000000011 txs=109 build_time=-2562047h47m16.854s insert_time=2562047h47m16.854s total_time=1.112s mgas=6.124781 mgasps=5.507383019267266 |  
  |   | 2025-05-29 08:44:52.216 | 2025-05-28T23:44:52.21003882Z stdout F t=2025-05-28T23:44:52+0000 lvl=info msg="Scheduled sequencer action" delta=-19.735µs |  
  |   | 2025-05-29 08:44:52.216 | 2025-05-28T23:44:52.210071399Z stdout F t=2025-05-28T23:44:52+0000 lvl=info msg="Started sequencing new block" parent=0x3d55356dfd639d4dbf7ec48063b728cedb48a3986cd4ed30841e40c490c0949d:13387132 l1Origin=0xdbda85dee62dd85cd7d314fb171bb8652563e4767dacbb31ada84583dff5f13d:8428128 |  
  |   | 2025-05-29 08:44:52.216 | 2025-05-28T23:44:52.213437966Z stdout F t=2025-05-28T23:44:52+0000 lvl=info msg="Scheduled sequencer action" delta=736.658505ms |  
  |   | 2025-05-29 08:44:52.965 | 2025-05-28T23:44:52.953488889Z stdout F t=2025-05-28T23:44:52+0000 lvl=info msg="Sequencer sealed block" payloadID=0x035cbcbb51ca9a73 block=0x9b0e681af1f546d9dffd3e2180f6861fdc0cbd5d4d689d3ec67d036095af2d26:13387133 parent=0x3d55356dfd639d4dbf7ec48063b728cedb48a3986cd4ed30841e40c490c0949d:13387132 txs=138 time=1748475893 |  
  |   | 2025-05-29 08:44:52.965 | 2025-05-28T23:44:52.953519027Z stdout F t=2025-05-28T23:44:52+0000 lvl=info msg="Sequencer inserted block" block=0x9b0e681af1f546d9dffd3e2180f6861fdc0cbd5d4d689d3ec67d036095af2d26:13387133 parent=0x3d55356dfd639d4dbf7ec48063b728cedb48a3986cd4ed30841e40c490c0949d:13387132 |  
  |   | 2025-05-29 08:44:52.965 | 2025-05-28T23:44:52.953533371Z stdout F t=2025-05-28T23:44:52+0000 lvl=info msg="Publishing signed execution payload on p2p" id=0x9b0e681af1f546d9dffd3e2180f6861fdc0cbd5d4d689d3ec67d036095af2d26:13387133 |  
  |   | 2025-05-29 08:44:52.965 | 2025-05-28T23:44:52.961297957Z stdout F t=2025-05-28T23:44:52+0000 lvl=info msg="Inserted new L2 unsafe block" hash=0x9b0e681af1f546d9dffd3e2180f6861fdc0cbd5d4d689d3ec67d036095af2d26 number=13387133 state_root=0x5ad1e3a821a3ad037514cb5c1e3a6669408183f6e1a74801ac29efda10609452 timestamp=1748475893 parent=0x3d55356dfd639d4dbf7ec48063b728cedb48a3986cd4ed30841e40c490c0949d prev_randao=0x26dc26d2b58c4cbf0e9b719eca40ce18546c1d3b00f1261d61c32373dd9ff2cf fee_recipient=0x4200000000000000000000000000000000000011 txs=138 build_time=-2562047h47m16.854s insert_time=2562047h47m16.854s total_time=751.079ms mgas=9.408709 mgasps=12.526910207087813 |  
  |   | 2025-05-29 08:44:52.965 | 2025-05-28T23:44:52.961365275Z stdout F t=2025-05-28T23:44:52+0000 lvl=info msg="Sequencer paused until new events"

Execution client side (reth):

2025-05-29 08:44:52.291 | 2025-05-28T23:44:52.193829913Z stdout F 2025-05-28T23:44:52.193805Z  INFO reth_node_events::node: Block added to canonical chain number=13387132 hash=0x3d55356dfd639d4dbf7ec48063b728cedb48a3986cd4ed30841e40c490c0949d peers=0 txs=109 gas=6.12 Mgas gas_throughput=634.30 Ggas/second full=0.0% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=9.656µs |  
-- | -- | --
  |   | 2025-05-29 08:44:52.291 | 2025-05-28T23:44:52.193841104Z stdout F 2025-05-28T23:44:52.193825Z  INFO reth_node_events::node: Canonical chain committed number=13387132 hash=0x3d55356dfd639d4dbf7ec48063b728cedb48a3986cd4ed30841e40c490c0949d elapsed=110.818µs |  
  |   | 2025-05-29 08:44:52.291 | 2025-05-28T23:44:52.213045784Z stdout F 2025-05-28T23:44:52.213003Z  INFO reth_payload_builder::service: New payload job created id=0x035cbcbb51ca9a73 parent=0x3d55356dfd639d4dbf7ec48063b728cedb48a3986cd4ed30841e40c490c0949d |  
  |   | 2025-05-29 08:44:53.043 | 2025-05-28T23:44:52.961051624Z stdout F 2025-05-28T23:44:52.960997Z  INFO reth_node_events::node: Block added to canonical chain number=13387133 hash=0x9b0e681af1f546d9dffd3e2180f6861fdc0cbd5d4d689d3ec67d036095af2d26 peers=0 txs=138 gas=9.41 Mgas gas_throughput=1511.92 Ggas/second full=0.0% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=6.223µs |  
  |   | 2025-05-29 08:46:16.707 | 2025-05-28T23:46:16.508357003Z stdout F 2025-05-28T23:46:16.508292Z  INFO reth::cli: Status connected_peers=0 latest_block=13387132 |  
  |   | 2025-05-29 08:46:41.749 | 2025-05-28T23:46:41.54515731Z stdout F 2025-05-28T23:46:41.545092Z  INFO reth::cli: Status connected_peers=0 latest_block=13387132 |  
  |   | 2025-05-29 08:47:07.048 | 2025-05-28T23:47:06.914201631Z stdout F 2025-05-28T23:47:06.914149Z  INFO reth::cli: Status connected_peers=0 latest_block=13387132 |  
  |   | 2025-05-29 08:47:32.344 | 2025-05-28T23:47:32.235507563Z stdout F 2025-05-28T23:47:32.235451Z  INFO reth::cli: Status connected_peers=0 latest_block=13387132 |  
  |   | 2025-05-29 08:47:57.896 | 2025-05-28T23:47:57.750805134Z stdout F 2025-05-28T23:47:57.750750Z  INFO reth::cli: Status connected_peers=0 latest_block=13387132 |  
  |   | 2025-05-29 08:48:23.194 | 2025-05-28T23:48:23.064401059Z stdout F 2025-05-28T23:48:23.064330Z  INFO reth::cli: Status connected_peers=0 latest_block=13387132 |  
  |   | 2025-05-29 08:48:48.241 | 2025-05-28T23:48:48.083621931Z stdout F 2025-05-28T23:48:48.083538Z  INFO reth::cli: Status connected_peers=0 latest_block=13387132 |  
  |   | 2025-05-29 08:48:50.745 | 2025-05-28T23:48:50.591658267Z stdout F 2025-05-28T23:48:50.591611Z  INFO reth_node_events::node: Canonical chain committed number=13387133 hash=0x9b0e681af1f546d9dffd3e2180f6861fdc0cbd5d4d689d3ec67d036095af2d26 elapsed=160.826µs

As you can see, there's a significant delay between the get_payload call and the subsequent FCU (without attr) call for block 13387133.

My question is:

What are the possible causes for the sequencer to enter the "paused until new events" state?

[joohhnnn]

Hey, when you see "Sequencer paused until new events", it basically means the sequencer is waiting for something to happen. Here's what's probably going on:

1. If you're running as a verifier (without --sequencer.enabled) or lost leader status in op-conductor, that's why it's paused.

2. Most likely, it's waiting for reth to respond. After the sequencer asks reth to build a block, it has to wait for a response. If reth is busy syncing or running out of memory, you'll see this message.

3. Sometimes reth throws temporary errors (like ErrEngineSyncing). The sequencer will try again in a few seconds.

4. If you set --sequencer-max-safe-lag, it might pause when the unsafe head gets too far ahead of the safe head.

To figure out what's happening:

• Look at the logs right before the "paused" message
• Try --log.level=debug to see more details
• Check if reth is actually processing payloads
• If you're using conductor, make sure it hasn't switched leaders
• If you set max-safe-lag, check the gap between unsafe and safe heads

The sequencer should start working again once reth responds or you get a new L1 block.

[0xNarumi]

Thanks for the answer. I have one follow-up question.

As you can see in the op-node logs above, the Inserted new L2 unsafe block log was emitted — which suggests that the forkchoiceUpdated call to the execution engine was successful.

So where could the process possibly be hanging after that?

[opfocus]

Are you already investigating the possibility of a hardware performance issue?

[0xNarumi]

I haven’t looked into it deeply yet, but I doubt it’s a hardware performance issue — we’re using a pretty beefy machine.
Do you happen to know what kind of hardware-related performance issues could potentially cause this?

[opfocus]

Now can you still observe the delay happening?

[opfocus]

From the logs, there does appear to be a noticeable delay. Please investigate the causes mentioned above.

[opfocus]

We noticed that reth has a fixed version. This may not be relevant to the question you mentioned, but I feel it may still be worth paying attention to https://github.com/paradigmxyz/reth/releases/tag/v1.4.7