Skip to content

Add out-of-energy error to db logs - #6612

Open
aasoni wants to merge 4 commits into
masterfrom
alessandro/report-out-of-energy-errors-in-module-logs
Open

aasoni wants to merge 4 commits into
masterfrom
alessandro/report-out-of-energy-errors-in-module-logs

Conversation

@aasoni

@aasoni aasoni commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Description of Changes

Whenever we abort a reducer because it used the maximum allowed cpu usage (fuel) the database logs don't show anything.
This means that the reason for the error is completely hidden to end users who don't have access to host logs.
This PR add that error to the database logs.

API and ABI breaking changes

No

Rollback safety impact

n/a

Expected complexity level and risk

1

Testing

I didn't test this, but the code is very simple so I believe compilation and existing unit tests are sufficient.

@aasoni
aasoni requested a review from Shubham8287 November 17, 2026 10:50

@lisandroct lisandroct left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have other comments/concerns besides the other two:

  • Claude reported that with these changes we would be getting duplicated reports in V8: "v8/error.rs:617 already writes the JS error and its stack trace to the module logs at Panic level, and this adds a second Error-level entry for the same failure."
  • I believe we should also handle views and report this same out of energy error there. Views also call handle_outer_error, so maybe we can move the report over there?
  • I'm not super familiar with all of our errors and just based on the name OutOfEnergy I wonder, do we throw this error if the user's energy balance is low too? If we do, the message "maximum CPU usage allowed per reducer call" would not be the appropriate one.

"reducer aborted after {:.2?}: {reason}",
result.stats.timings.total_duration
);
log_reducer_error(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're calling handle_outer_error (in line 1245) which increments WORKER_METRICS.wasm_instance_errors. We were already doing that.
Now we're calling log_reducer_error here which increments WORKER_METRICS.sender_errors.
So we will be incrementing two different error counters, which is probably not what we want.

);
log_reducer_error(
inst.replica_ctx(),
timestamp,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we use InstanceEnv::now_for_logging() here?

@aasoni

aasoni commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

@lisandroct
The other suggestions are good for this:

  • I'm not super familiar with all of our errors and just based on the name OutOfEnergy I wonder, do we throw this error if the user's energy balance is low too? If we do, the message "maximum CPU usage allowed per reducer call" would not be the appropriate one.

The energy system that exists on the website to track people's usage is unrelated to this fuel limit. This is an old system in place to prevent reducers from accidentally (or intentionally) running for too long. It's not great, but right now it's also silent which is even worse.

@aasoni
aasoni force-pushed the alessandro/report-out-of-energy-errors-in-module-logs branch from 536bc43 to e7f9380 Compare November 19, 2026 10:16

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants