Skip to content

[benchmarks] overhaul benchmarks#11565

Merged
sayakpaul merged 91 commits intomainfrom
benchmarking-overhaul
Jul 4, 2025
Merged

[benchmarks] overhaul benchmarks#11565
sayakpaul merged 91 commits intomainfrom
benchmarking-overhaul

Conversation

@sayakpaul
Copy link
Member

@sayakpaul sayakpaul commented May 16, 2025 •
edited
Loading

What does this PR do?

This PR considerably simplifies how we do benchmarks. Instead of using entire pipeline-level benchmarks across different tasks, we will now ONLY benchmark the diffusion network that is the most compute-intensive part in a standard diffusion workflow.

To make the estimates more realistic, we will make use of pre-trained checkpoints and dummy inputs with reasonable dimensionalities.

I ran benchmarking_flux.py on an 80GB A100 on a batch size of 1 and got the following results:

image

Analyze the results in this Space: https://huggingface.co/spaces/diffusers/benchmark-analyzer

By default, all benchmarks will use a batch size of 1, eliminating CFG.

How to add your benchmark?

Adding benchmarks for a new model class (SanaTransformer2DModel, for example) boils down to the following:

  1. Define the dummy inputs of the model.
  2. Define the benchmarking scenarios we should run the benchmark on.

This is what benchmarking_flux.py does. More modularization can be shipped afterward.

Idea would be to merge this PR with pre-configured benchmarks for a few popular models and open others to the community.

TODOs

Utilities:

  • To fire the execution of the individual model-level benchmarks sequentially.
  • To combine CSVs from multiple different model classes.
  • Central dataset update and Slack notification.

@DN6 could you give the approach a quick look? I can then work on resolving the TODOs.

sayakpaul
sayakpaul commented May 16, 2025
View reviewed changes
benchmarks/benchmarking_flux.py


if __name__ == "__main__":
scenarios = [
Copy link
Member Author

@sayakpaul sayakpaul May 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Covered the following scenarios:

  • Regular BF16 with compilation
  • NF4
  • Layerwise upcasting
  • Group offloading

@sayakpaul
Copy link
Member Author

sayakpaul commented May 20, 2025 •
edited
Loading

Added SDXL, Wan (14B), and LTX (13B) on top of Flux:

Results
scenario model_cls num_params_M flops_M time_plain_s mem_plain_GB time_compile_s mem_compile_GB fullgraph mode
0 Wan-AI/Wan2.1-T2V-14B-Diffusers-bf16 WanTransformer3DModel 14288.5 7.85612e+08 10.797 31.17 8.974 31.77 1 default
1 Wan-AI/Wan2.1-T2V-14B-Diffusers-layerwise-upcasting WanTransformer3DModel 14288.5 7.85612e+08 10.702 26.78 nan nan nan nan
2 Wan-AI/Wan2.1-T2V-14B-Diffusers-group-offload-leaf WanTransformer3DModel 14288.5 7.85612e+08 10.83 4.48 nan nan nan nan
3 stabilityai/stable-diffusion-xl-base-1.0-bf16 UNet2DConditionModel 2567.46 5.9791e+06 0.085 5.05 0.058 5.39 1 default
4 stabilityai/stable-diffusion-xl-base-1.0-layerwise-upcasting UNet2DConditionModel 2567.46 5.9791e+06 0.175 4.89 nan nan nan nan
5 stabilityai/stable-diffusion-xl-base-1.0-group-offload-leaf UNet2DConditionModel 2567.46 5.9791e+06 0.383 0.2 nan nan nan nan
6 black-forest-labs/FLUX.1-dev-bf16 FluxTransformer2DModel 11901.4 5.95295e+07 0.535 22.61 0.388 22.85 1 default
7 black-forest-labs/FLUX.1-dev-bnb-nf4 FluxTransformer2DModel 5952.25 17263.8 0.574 6.7 nan nan nan nan
8 black-forest-labs/FLUX.1-dev-layerwise-upcasting FluxTransformer2DModel 11901.4 5.95295e+07 0.621 22.18 nan nan nan nan
9 black-forest-labs/FLUX.1-dev-group-offload-leaf FluxTransformer2DModel 11901.4 5.95295e+07 1.536 0.53 nan nan nan nan
10 Lightricks/LTX-Video-0.9.7-dev-bf16 LTXVideoTransformer3DModel 13042.6 1.67583e+08 1.446 25.21 1.137 25.63 1 default
11 Lightricks/LTX-Video-0.9.7-dev-layerwise-upcasting LTXVideoTransformer3DModel 13042.6 1.67583e+08 1.529 24.38 nan nan nan nan
12 Lightricks/LTX-Video-0.9.7-dev-group-offload-leaf LTXVideoTransformer3DModel 13042.6 1.67583e+08 1.917 1.04 nan nan nan nan

@sayakpaul sayakpaul marked this pull request as ready for review May 20, 2025 11:08
@sayakpaul sayakpaul changed the title [WIP][benchmarks] overhaul benchmarks [benchmarks] overhaul benchmarks May 20, 2025
@sayakpaul
Copy link
Member Author

sayakpaul commented May 20, 2025

Cc: @a-r-r-o-w if you want to add some caching benchmarks (in a later PR), I think that would be really great!

@sayakpaul sayakpaul requested a review from DN6 May 20, 2025 12:09
@sayakpaul
Copy link
Member Author

sayakpaul commented May 20, 2025 •
edited
Loading

@DN6 this is ready for a review.

This is how the final CSV for this stage looks like:
https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/collated_results.csv

I have confirmed in this run that it works as expected:
https://github.com/huggingface/diffusers/actions/runs/15138495257/job/42570011907

@sayakpaul
Copy link
Member Author

sayakpaul commented Jun 23, 2025

@DN6 a gentle ping.

DN6
DN6 reviewed Jul 2, 2025
View reviewed changes
.github/workflows/benchmark.yml Outdated
workflow_dispatch:
schedule:
- cron: "30 1 1,15 * *" # every 2 weeks on the 1st and the 15th of every month at 1:30 AM
- cron: "0 17 * * 1" # every monday at 5 PM.
Copy link
Collaborator

@DN6 DN6 Jul 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a blocker. But why run every week? Is a monthly benchmark not sufficient?

Copy link
Member Author

@sayakpaul sayakpaul Jul 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True. Changing to bi-weekly.

@sayakpaul
Copy link
Member Author

sayakpaul commented Jul 2, 2025

@anijain2305 just a ping to let you know that we're merging this PR which will run the benchmarking suite bi-weekly and report the results here: https://huggingface.co/datasets/diffusers/benchmarks/blob/main/collated_results.csv

@anijain2305
Copy link
Contributor

anijain2305 commented Jul 2, 2025

Thanks for setting this up. This will be really helpful for tracking progress and identifying regression

sayakpaul
sayakpaul commented Jul 3, 2025
View reviewed changes
McPatate
McPatate reviewed Jul 3, 2025
View reviewed changes
@sayakpaul
Copy link
Member Author

sayakpaul commented Jul 4, 2025

Since everything is passing now, will merge this PR :)
https://github.com/huggingface/diffusers/actions/runs/16065987231/job/45340516693

@sayakpaul sayakpaul merged commit e6639fe into main Jul 4, 2025
12 checks passed
@sayakpaul sayakpaul deleted the benchmarking-overhaul branch July 4, 2025 05:34
@sayakpaul
Copy link
Member Author

sayakpaul commented Jul 4, 2025

Also cc @a-r-r-o-w for #11565 (comment) (not urgent, when you get time).

tolgacangoz pushed a commit to tolgacangoz/diffusers that referenced this pull request Jul 5, 2025
@sayakpaul @DN6
@McPatate @tolgacangoz
[benchmarks] overhaul benchmarks (huggingface#11565)
5e8c8e2 
* start overhauling the benchmarking suite.

* fixes

* fixes

* checking.

* checking

* fixes.

* error handling and logging.

* add flops and params.

* add more models.

* utility to fire execution of all benchmarking scripts.

* utility to push to the hub.

* push utility improvement

* seems to be working.

* okay

* add torchprofile dep.

* remove total gpu memory

* fixes

* fix

* need a big gpu

* better

* what's happening.

* okay

* separate requirements and make it nightly.

* add db population script.

* update secret name

* update secret.

* population db update

* disable db population for now.

* change to every monday

* Update .github/workflows/benchmark.yml

Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>

* quality improvements.

* reparate hub upload step.

* repository

* remove csv

* check

* update

* update

* threading.

* update

* update

* updaye

* update

* update

* update

* remove peft dep

* upgrade runner.

* fix

* fixes

* fix merging csvs.

* push dataset to the Space repo for analysis.

* warm up.

* add a readme

* Apply suggestions from code review

Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>

* address feedback

* Apply suggestions from code review

* disable db workflow.

* update to bi weekly.

* enable population

* enable

* updaye

* update

* metadata

* fix

---------

Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DN6 DN6 DN6 approved these changes

@McPatate McPatate McPatate left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@sayakpaul @a-r-r-o-w @McPatate @anijain2305 @DN6