Sequencer failed to send the transaction to the da layer

[MartinOndejka]

Error log:

("Could not send query to da node after 1 tries. The process may not be running, please check the daemon-argument"
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]:      ((host_and_port (127.0.0.1 1924)) (daemon-argument da-node-0))
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]:      ((rpc_error
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]:        (Uncaught_exn
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]:         ((location "server-side rpc computation")
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]:          (exn
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]:           (monitor.ml.Error
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]:            "Get_diff_source exception: Diff not found for ledger hash 2860702436500467301993831580416896475558048128111658825203583070728367310837"
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]:            ("Raised at Base__Error.raise in file \"src/error.ml\" (inlined), line 8, characters 14-30"
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]:             "Called from Base__Option.value_exn in file \"src/option.ml\", line 136, characters 4-21"
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]:             "Called from Da_layer__Node.implementations.(fun) in file \"src/app/zeko/da_layer/lib/node.ml\", line 340, characters 15-352"
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]:             "Called from Base__Result.map in file \"src/result.ml\", line 122, characters 19-24"
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]:             "Called from Base__Monad.Make_general.Monad_infix.(>>|) in file \"src/monad.ml\" (inlined), line 34, characters 22-30"
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]:             "Called from Async_rpc_kernel__Implementations.Instance.apply_implementation.(fun) in file \"src/implementations.ml\", line 498, characters 25-67"
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]:             "Called from Base__Result.try_with in file \"src/result.ml\", line 195, characters 9-15"))))))
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]:       (connection_description <created-directly>) (rpc_tag Get_diff_source)
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]:       (rpc_version 1))))
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]:    ("Raised at Base__Error.raise in file \"src/error.ml\" (inlined), line 8, characters 14-30"
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]:     "Called from Sequencer_lib__Zeko_sequencer.Make.create.(fun) in file \"src/app/zeko/sequencer/lib/zeko_sequencer.ml\", line 910, characters 10-23"
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]:     "Called from Async_kernel__Deferred0.bind.(fun) in file \"src/deferred0.ml\", line 54, characters 64-69"
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]:     "Called from Async_kernel__Job_queue.run_job in file \"src/job_queue.ml\" (inlined), line 128, characters 2-5"
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]:     "Called from Async_kernel__Job_queue.run_jobs in file \"src/job_queue.ml\", line 169, characters 6-47"
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]:     "Caught by monitor block_on_async"))
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]: Raised at Base__Result.ok_exn in file "src/result.ml" (inlined), line 201, characters 17-26
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]: Called from Async_unix__Thread_safe.block_on_async_exn in file "src/thread_safe.ml", line 132, characters 29-63
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]: Called from Async_unix__Thread_safe.block_on_async_exn in file "src/thread_safe.ml" (inlined), line 181, characters 27-54
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]: Called from Dune__exe__Run.run in file "src/app/zeko/sequencer/run.ml", line 19, characters 4-439
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]: Called from Core_kernel__Command.For_unix.run.(fun) in file "src/command.ml", line 2453, characters 8-238
Oct 30 12:12:55 zeko c2s13gp7zdpv7xgm9d3c03xr6jkmj9px-sequencer[106479]: Called from Base__Exn.handle_uncaught_aux in file "src/exn.ml", line 111, characters 6-10

[L-as]

I think the error message is wrong. Inside it says it couldn't find diff for ledger hash.

[MartinOndejka]

Oh right, the error message is correct, my assumption with connection issue was wrong. The message came from node, so connection was fine. My guess is that the sequencer applied the transaction and just before it was able to send transaction to da, it crashed. There is a queue of transactions to be sent to the da layer, because the order is important and we can't wait with the mask application until the transaction has been sent.

[L-as]

No, it says ("Could not send query to da node after 1 tries. The process may not be running, please check the daemon-argument", but that's not what it says inside.

[L-as]

There's no connection issue.

[MartinOndejka]

Fix: Store the state of the da queue of diffs and on restart requeue them.