Question about energy estimate gap between ecologits and new study

[cleophass]

Hello Ecologits team,

I recently came across the study "How Hungry is AI? Benchmarking Energy, Water, and Carbon Footprint of LLM Inference" (arXiv:2505.09598v2). The paper introduces an infrastructure-aware benchmarking framework for inference footprint estimation across multiple proprietary and open-source LLMs, including GPT-4o.

According to their benchmarks, GPT-4o consumes approximately 1.788 Wh for long prompts and 0.42 Wh for short ones (as defined in their methodology). This seems considerably lower than the estimates produced by your calculator, which for similar prompt sizes (e.g., 100 input / 300 output) estimates up to 35.1 Wh.

At first, the methodologies appear structurally similar — both multiply execution time by a power value — but the paper models things like:

• GPU utilization rate and batch size
• Number of active GPUs based on model class
• Data center overhead (PUE)
• Inference time

As I dove deeper into their methodology, I realized that their energy estimation, like yours, is based on inference time. However, instead of relying on leaderboard studies or static power assumptions, they focus on GPU power draw and also factor in batch size and dynamic GPU utilization to compute the per-query energy share.

Have you had the chance to review this study? I’d be very interested in your perspective on the methodology and whether integrating infrastructure-aware parameters (e.g., TPS, dynamic utilization, or model class) is something you’re considering in future versions.

Thank you in advance for your time, and again for your contributions to the field.

[samuelrince]

Hello @cleophass,

Thank you for your feedback! We are aware that currently, our methodology might overshoot energy consumption. We are launching a workshop this summer to update parts of the methodology to fix that!

The issue comes mostly from the energy benchmark we use that is far from a "large-scale production deployment" scenario. We'll let you know when that's available!

PS: Just in case, if you are interested in contributing to this and have some time, I highly encourage you to join us 🤗

[cleophass]

great news for your workshop, can't wait to see the results!

Otherwise, as a feature idea, I was thinking of adding a method for estimation that doesn't rely on an API call — a kind of method that would take into account the generated tokens, latency time, model, and hosting location. This could be useful for benchmarking without making API calls. would you be interested if I open a PR for this?

[adrienbanse]

Hey @cleophass, we're currently working on such features! :)