LLM Inference Methodology: questions on energy required to train models initially

[ketsapiwiq]

Hi!

The question I have is: do you have plans for integrating an estimation of the foundational models' initial training costs in terms of energy? I think there is some data for models like LLama 3?

Thank you so much for your work, we're using it for a chatbot arena project (much like https://chat.lmsys.org/) that compares models to improve text-based LLMs on the French language and other languages from France called LANGU:IA (I can send you a private link if you're interested to see it in action).

Have a nice day!

[samuelrince]

Hello @ketsapiwiq,

It is indeed an excellent question, thanks for asking! We do not plan (in the short term, at least) to support training impacts in EcoLogits. The methodology is designed to assess impacts at inference and is intended to address big providers like OpenAI. To include a part of the training impacts into each request, we would need "trade secret" information from these providers about the training impacts but also the usage of their services.

For instance, Meta computed the energy consumption and carbon emissions (1900 tCO2eq) (only derived from direct energy consumption) for Llama 3 70B source. If we were to include it in our methodology for inference, we would somehow need to divide this impact among all the requests that will ever be made to this model. But even estimating this number is very difficult.

Plus, we quickly realize that for models deployed at that scale, dividing ~2000 tCO2eq or even 10,000 tCO2eq by billions of requests made by millions of users is negligible compared to the impact of simply running the model.

What we can consider adding in the future is a customizable way of accounting for the training impacts if you know your own usage as a company, for instance, but it's not really on the roadmap for now. ;)

Thank you so much for your work, we're using it for a chatbot arena project (much like chat.lmsys.org) that compares models to improve text-based LLMs on the French language and other languages from France called LANGU:IA (I can send you a private link if you're interested to see it in action).

I am intrigued indeed, I'd be happy to see a live version of it! :)

[ketsapiwiq]

Thanks for this complete answer.
I understand dividing the training impact by the number of inference requests makes little sense, thanks for the insight.
Maybe if there is a clearer methodology for evaluating the initial training impact for other models beyond llama3-70b (hours of compute or maybe even based on the "n trillions of token for m epochs" info?) we could consider adding it as a static info.
But if the aim is to raise awareness to the (growing) inference costs, I agree it's not that important to focus on this number.
I sent you a link to hello@sento.re for the arena :)

[samuelrince]

Yes, I think that we will definitely try to integrate training impacts at some point when we are confident that our methodology covers it well. We have some other projects focused on it at Boavizta too. :)