fix: remove unnecessary wakes

[akundaz]

Background

When running unichain-builder in staging I saw excessive context switching (~60k/sec). Looking into the pipeline API I noticed that the PipelineExecutor made a call to cx.waker().wake_by_ref() in the manual Future::poll implementation, which seemed the likely culprit. That call tells the tokio executor to immediately poll the same future again, but since the cursor is not ready to advance yet, we end up going to that wake line very quickly because we don't do anything. This causes a hot loop in the tokio executor leading to heavy context switching.

However, simply removing the cx.waker().wake_by_ref() calls was not sufficient. What then ended up happening was that the tokio executor got starved and other tasks were severely delayed, which was caught by unichain-builder integration tests. So in addition to removing the wake calls I added in a yield point.

Changes

PipelineExecutor refactor

• Replace manual poll state machine with native async/await
• Remove Cursor enum and associated state management, this removed a lot of lines
• Remove unnecessary Provider generic from PayloadJob

The execution flow is now:

• PipelineExecutor::run() → creates executor and boxed future
• execute() → orchestrates init → steps → finalize
• run_steps() → async loop over pipeline steps

Cooperative yielding: Added a tokio::task::yield_now().await call after each step transition to prevent executor starvation. I didn't investigate what the best placement for this would be, but this spot works for the unichain-builder tests I was checking.

[julio4]

Nice good find on this hot reschedule loop! Just one issue that should fix tests as well and should be good to go