Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved performance benchmark #168

Open
wants to merge 17 commits into
base: main
Choose a base branch
from
53 changes: 41 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -248,27 +248,56 @@ Sample Jab usage in console application can be found in [src/samples/ConsoleSamp

The performance benchmark project is available in [src/Jab.Performance/](src/Jab.Performance/).

And the results in [docs/benchmark/](docs/benchmark/)

### Startup time

The startup time benchmark measures time between application startup and the first service being resolved.

```
| Method | Mean | Error | StdDev | Ratio | RatioSD | Gen 0 | Gen 1 | Gen 2 | Allocated |
|------- |------------:|----------:|----------:|-------:|--------:|-------:|-------:|------:|----------:|
| MEDI | 2,437.88 ns | 14.565 ns | 12.163 ns | 220.91 | 2.72 | 0.6332 | 0.0114 | - | 6632 B |
| Jab | 11.03 ns | 0.158 ns | 0.123 ns | 1.00 | 0.00 | 0.0046 | - | - | 48 B |
```
| Method | Mean | Error | StdDev | Gen0 | Gen1 | Allocated |
|--------------- |-------------:|--------------:|------------:|-------:|-------:|----------:|
| Jab_Singleton | 8.763 ns | 0.5496 ns | 0.0301 ns | 0.0067 | - | 56 B |
| MEDI_Singleton | 1,883.539 ns | 1,976.8621 ns | 108.3585 ns | 0.8640 | 0.2155 | 7232 B |
| | | | | | | |
| Jab_Scoped | 8.445 ns | 6.0425 ns | 0.3312 ns | 0.0038 | - | 32 B |
| MEDI_Scoped | 1,894.341 ns | 1,854.7252 ns | 101.6637 ns | 0.8640 | 0.2155 | 7232 B |
| | | | | | | |
| Jab_Transient | 8.695 ns | 11.2536 ns | 0.6168 ns | 0.0038 | - | 32 B |
| MEDI_Transient | 1,894.840 ns | 2,000.5040 ns | 109.6544 ns | 0.8640 | 0.2155 | 7232 B |
| | | | | | | |
| Jab_Mixed | 9.722 ns | 2.2865 ns | 0.1253 ns | 0.0067 | - | 56 B |
| MEDI_Mixed | 2,454.882 ns | 2,626.4526 ns | 143.9647 ns | 1.0834 | 0.2689 | 9064 B |
| | | | | | | |
| Jab_Complex | 13.402 ns | 4.5401 ns | 0.2489 ns | 0.0067 | - | 56 B |
| MEDI_Complex | 2,350.227 ns | 2,335.5694 ns | 128.0204 ns | 1.1330 | 0.2823 | 9496 B |

### GetService

The `GetService` benchmark measures the `provider.GetService<IService>()` call.

```
| Method | Mean | Error | StdDev | Ratio | RatioSD | Gen 0 | Gen 1 | Gen 2 | Allocated |
|------- |----------:|----------:|----------:|------:|--------:|-------:|------:|------:|----------:|
| MEDI | 39.340 ns | 0.2419 ns | 0.2263 ns | 7.01 | 0.09 | 0.0023 | - | - | 24 B |
| Jab | 5.619 ns | 0.0770 ns | 0.0643 ns | 1.00 | 0.00 | 0.0023 | - | - | 24 B |
```
### Singleton

| Method | Mean | Error | StdDev | Ratio | RatioSD |
|-------- |-------------:|--------------:|--------------:|---------:|---------:|
| **Jab** | **3.325 ns** | **2.1053 ns** | **0.1154 ns** | **1.00** | **0.00** |
| MEDI | 9.241 ns | 0.2836 ns | 0.0155 ns | 2.78 | 0.09 |

### Transient

| Method | Mean | Error | StdDev | Ratio | RatioSD |
|-------- |-------------:|-------------:|-------------:|---------:|---------:|
| **Jab** | **11.29 ns** | **3.599 ns** | **0.197 ns** | **1.00** | **0.00** |
| MEDI | 13.46 ns | 1.805 ns | 0.099 ns | 1.19 | 0.03 |


### Complex

| Method | Mean | Error | StdDev | Ratio | RatioSD |
|-------- |-------------:|--------------:|------------:|----------:|---------:|
| **Jab** | **279.6 ns** | **154.26 ns** | **8.46 ns** | **1.00** | **0.00** |
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is quite interesting. Do you have an idea why complex resolution is so much slower in Jab?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to find it and didn't understand why yet.
Some could be explained as the lock {}, but I removed it and didn't see a big improvement. See #169

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I managed to beat the MEDI performance for this case.

  • Removing locks and using LazyInitializer.EnsureInitialized
  • Removing the TryAddDisposable() - (this could be done in compilation time instead of runtime)
  • Replace all this.GetService<T>() for ((IServiceProvider<T>)this).GetService()
Method NumbersOfCalls NumbersOfClasses Mean Error StdDev Ratio RatioSD Gen0 Gen1 Allocated Alloc Ratio
Jab 1 1 270.81 ns 63.19 ns 3.463 ns 1.00 0.00 0.0563 - 472 B 1.00
Improved_Jab 1 1 92.74 ns 65.84 ns 3.609 ns 0.34 0.01 0.0736 - 616 B 1.31
MEDI 1 1 157.47 ns 37.13 ns 2.035 ns 0.58 0.01 0.0870 0.0002 728 B 1.54
Jab 1 2 543.88 ns 229.26 ns 12.566 ns 1.00 0.00 0.1059 - 888 B 1.00
Improved_Jab 1 2 255.73 ns 213.70 ns 11.714 ns 0.47 0.03 0.1402 - 1176 B 1.32
MEDI 1 2 276.63 ns 129.49 ns 7.098 ns 0.51 0.02 0.1364 0.0005 1144 B 1.29
Jab 1 3 814.39 ns 65.88 ns 3.611 ns 1.00 0.00 0.1554 - 1304 B 1.00
Improved_Jab 1 3 389.66 ns 204.35 ns 11.201 ns 0.48 0.01 0.2074 0.0010 1736 B 1.33
MEDI 1 3 397.50 ns 134.14 ns 7.353 ns 0.49 0.01 0.1864 0.0010 1560 B 1.20
Jab 10 1 2,811.38 ns 127.55 ns 6.992 ns 1.00 0.00 0.5608 - 4720 B 1.00
Improved_Jab 10 1 932.41 ns 132.33 ns 7.253 ns 0.33 0.00 0.7362 - 6160 B 1.31
MEDI 10 1 1,517.62 ns 697.80 ns 38.249 ns 0.54 0.01 0.8698 0.0019 7280 B 1.54
Jab 10 2 5,394.83 ns 1,473.69 ns 80.778 ns 1.00 0.00 1.0605 - 8880 B 1.00
Improved_Jab 10 2 2,563.65 ns 542.04 ns 29.711 ns 0.48 0.01 1.4038 - 11760 B 1.32
MEDI 10 2 2,820.40 ns 1,988.28 ns 108.984 ns 0.52 0.03 1.3657 0.0038 11440 B 1.29
Jab 10 3 8,515.73 ns 1,780.40 ns 97.590 ns 1.00 0.00 1.5564 - 13040 B 1.00
Improved_Jab 10 3 3,791.29 ns 348.95 ns 19.127 ns 0.45 0.01 2.0752 0.0076 17360 B 1.33
MEDI 10 3 4,014.64 ns 1,245.71 ns 68.282 ns 0.47 0.00 1.8616 0.0076 15600 B 1.20
Jab 100 1 28,236.83 ns 12,613.72 ns 691.401 ns 1.00 0.00 5.6152 - 47200 B 1.00
Improved_Jab 100 1 10,499.51 ns 15,496.89 ns 849.437 ns 0.37 0.02 7.3547 - 61600 B 1.31
MEDI 100 1 16,673.16 ns 3,621.32 ns 198.497 ns 0.59 0.02 8.6975 0.0153 72800 B 1.54
Jab 100 2 56,434.78 ns 4,705.42 ns 257.920 ns 1.00 0.00 10.5591 - 88800 B 1.00
Improved_Jab 100 2 25,188.64 ns 6,705.33 ns 367.542 ns 0.45 0.00 14.0381 0.0305 117600 B 1.32
MEDI 100 2 27,463.55 ns 15,492.67 ns 849.206 ns 0.49 0.02 13.6719 0.0305 114400 B 1.29
Jab 100 3 84,041.73 ns 35,261.40 ns 1,932.796 ns 1.00 0.00 15.5029 - 130400 B 1.00
Improved_Jab 100 3 38,046.35 ns 11,730.31 ns 642.978 ns 0.45 0.02 20.7520 0.0610 173600 B 1.33
MEDI 100 3 42,855.80 ns 46,857.75 ns 2,568.432 ns 0.51 0.04 18.6157 0.0610 156000 B 1.20

| MEDI | 149.2 ns | 20.02 ns | 1.10 ns | 0.53 | 0.02 |



## Unity installation
1. Navigate to the Packages directory of your project.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
```

BenchmarkDotNet v0.13.10, Windows 10 (10.0.19045.3930/22H2/2022Update)
AMD Ryzen 9 3900XT, 1 CPU, 24 logical and 12 physical cores
.NET SDK 8.0.100
[Host] : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX2
ShortRun : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX2

Job=ShortRun IterationCount=3 LaunchCount=1
WarmupCount=3

```
| Method | NumbersOfCalls | NumbersOfClasses | Mean | Error | StdDev | Ratio | RatioSD | Gen0 | Gen1 | Allocated | Alloc Ratio |
|------- |--------------- |----------------- |------------:|--------------:|------------:|------:|--------:|--------:|-------:|----------:|------------:|
| **Jab** | **1** | **1** | **279.6 ns** | **154.26 ns** | **8.46 ns** | **1.00** | **0.00** | **0.0563** | **-** | **472 B** | **1.00** |
| MEDI | 1 | 1 | 149.2 ns | 20.02 ns | 1.10 ns | 0.53 | 0.02 | 0.0870 | 0.0002 | 728 B | 1.54 |
| | | | | | | | | | | | |
| **Jab** | **1** | **2** | **565.4 ns** | **80.38 ns** | **4.41 ns** | **1.00** | **0.00** | **0.1059** | **-** | **888 B** | **1.00** |
| MEDI | 1 | 2 | 269.8 ns | 103.70 ns | 5.68 ns | 0.48 | 0.01 | 0.1364 | 0.0005 | 1144 B | 1.29 |
| | | | | | | | | | | | |
| **Jab** | **1** | **3** | **829.7 ns** | **353.08 ns** | **19.35 ns** | **1.00** | **0.00** | **0.1554** | **-** | **1304 B** | **1.00** |
| MEDI | 1 | 3 | 384.2 ns | 25.62 ns | 1.40 ns | 0.46 | 0.01 | 0.1864 | 0.0010 | 1560 B | 1.20 |
| | | | | | | | | | | | |
| **Jab** | **10** | **1** | **2,846.8 ns** | **1,643.85 ns** | **90.10 ns** | **1.00** | **0.00** | **0.5608** | **-** | **4720 B** | **1.00** |
| MEDI | 10 | 1 | 1,419.2 ns | 93.91 ns | 5.15 ns | 0.50 | 0.02 | 0.8698 | 0.0019 | 7280 B | 1.54 |
| | | | | | | | | | | | |
| **Jab** | **10** | **2** | **5,674.5 ns** | **1,663.99 ns** | **91.21 ns** | **1.00** | **0.00** | **1.0605** | **-** | **8880 B** | **1.00** |
| MEDI | 10 | 2 | 2,742.9 ns | 1,163.85 ns | 63.79 ns | 0.48 | 0.02 | 1.3657 | 0.0038 | 11440 B | 1.29 |
| | | | | | | | | | | | |
| **Jab** | **10** | **3** | **7,961.3 ns** | **4,443.31 ns** | **243.55 ns** | **1.00** | **0.00** | **1.5564** | **-** | **13040 B** | **1.00** |
| MEDI | 10 | 3 | 3,895.1 ns | 1,591.50 ns | 87.24 ns | 0.49 | 0.02 | 1.8616 | 0.0076 | 15600 B | 1.20 |
| | | | | | | | | | | | |
| **Jab** | **100** | **1** | **29,641.7 ns** | **30,778.23 ns** | **1,687.06 ns** | **1.00** | **0.00** | **5.6152** | **-** | **47200 B** | **1.00** |
| MEDI | 100 | 1 | 14,530.2 ns | 7,006.36 ns | 384.04 ns | 0.49 | 0.01 | 8.6975 | 0.0153 | 72800 B | 1.54 |
| | | | | | | | | | | | |
| **Jab** | **100** | **2** | **54,178.3 ns** | **22,959.21 ns** | **1,258.47 ns** | **1.00** | **0.00** | **10.5591** | **-** | **88800 B** | **1.00** |
| MEDI | 100 | 2 | 26,824.2 ns | 4,858.13 ns | 266.29 ns | 0.50 | 0.01 | 13.6719 | 0.0305 | 114400 B | 1.29 |
| | | | | | | | | | | | |
| **Jab** | **100** | **3** | **92,719.6 ns** | **111,920.74 ns** | **6,134.75 ns** | **1.00** | **0.00** | **15.5029** | **-** | **130400 B** | **1.00** |
| MEDI | 100 | 3 | 39,650.9 ns | 26,436.59 ns | 1,449.08 ns | 0.43 | 0.01 | 18.6157 | 0.0610 | 156000 B | 1.20 |
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
```

BenchmarkDotNet v0.13.10, Windows 10 (10.0.19045.3930/22H2/2022Update)
AMD Ryzen 9 3900XT, 1 CPU, 24 logical and 12 physical cores
.NET SDK 8.0.100
[Host] : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX2
ShortRun : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX2

Job=ShortRun IterationCount=3 LaunchCount=1
WarmupCount=3

```
| Method | NumbersOfCalls | NumbersOfClasses | Mean | Error | StdDev | Ratio | RatioSD | Gen0 | Allocated | Alloc Ratio |
|------- |--------------- |----------------- |-------------:|-------------:|-----------:|------:|--------:|-------:|----------:|------------:|
| **Jab** | **1** | **1** | **25.09 ns** | **2.023 ns** | **0.111 ns** | **1.00** | **0.00** | **0.0105** | **88 B** | **1.00** |
| MEDI | 1 | 1 | 52.97 ns | 5.454 ns | 0.299 ns | 2.11 | 0.02 | 0.0220 | 184 B | 2.09 |
| | | | | | | | | | | |
| **Jab** | **1** | **2** | **48.33 ns** | **22.718 ns** | **1.245 ns** | **1.00** | **0.00** | **0.0172** | **144 B** | **1.00** |
| MEDI | 1 | 2 | 71.94 ns | 23.674 ns | 1.298 ns | 1.49 | 0.02 | 0.0286 | 240 B | 1.67 |
| | | | | | | | | | | |
| **Jab** | **1** | **3** | **66.67 ns** | **37.560 ns** | **2.059 ns** | **1.00** | **0.00** | **0.0238** | **200 B** | **1.00** |
| MEDI | 1 | 3 | 111.83 ns | 108.495 ns | 5.947 ns | 1.68 | 0.04 | 0.0353 | 296 B | 1.48 |
| | | | | | | | | | | |
| **Jab** | **10** | **1** | **259.18 ns** | **130.556 ns** | **7.156 ns** | **1.00** | **0.00** | **0.1049** | **880 B** | **1.00** |
| MEDI | 10 | 1 | 533.29 ns | 295.258 ns | 16.184 ns | 2.06 | 0.01 | 0.2193 | 1840 B | 2.09 |
| | | | | | | | | | | |
| **Jab** | **10** | **2** | **453.72 ns** | **208.118 ns** | **11.408 ns** | **1.00** | **0.00** | **0.1717** | **1440 B** | **1.00** |
| MEDI | 10 | 2 | 715.44 ns | 228.174 ns | 12.507 ns | 1.58 | 0.01 | 0.2861 | 2400 B | 1.67 |
| | | | | | | | | | | |
| **Jab** | **10** | **3** | **664.24 ns** | **323.259 ns** | **17.719 ns** | **1.00** | **0.00** | **0.2384** | **2000 B** | **1.00** |
| MEDI | 10 | 3 | 1,060.66 ns | 691.051 ns | 37.879 ns | 1.60 | 0.10 | 0.3529 | 2960 B | 1.48 |
| | | | | | | | | | | |
| **Jab** | **100** | **1** | **2,587.41 ns** | **1,476.947 ns** | **80.956 ns** | **1.00** | **0.00** | **1.0490** | **8800 B** | **1.00** |
| MEDI | 100 | 1 | 5,068.08 ns | 395.160 ns | 21.660 ns | 1.96 | 0.06 | 2.1973 | 18400 B | 2.09 |
| | | | | | | | | | | |
| **Jab** | **100** | **2** | **4,591.13 ns** | **2,233.109 ns** | **122.404 ns** | **1.00** | **0.00** | **1.7166** | **14400 B** | **1.00** |
| MEDI | 100 | 2 | 7,234.70 ns | 1,964.384 ns | 107.675 ns | 1.58 | 0.06 | 2.8687 | 24000 B | 1.67 |
| | | | | | | | | | | |
| **Jab** | **100** | **3** | **6,837.38 ns** | **2,789.230 ns** | **152.887 ns** | **1.00** | **0.00** | **2.3880** | **20000 B** | **1.00** |
| MEDI | 100 | 3 | 11,699.66 ns | 1,556.090 ns | 85.295 ns | 1.71 | 0.04 | 3.5248 | 29600 B | 1.48 |
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
```

BenchmarkDotNet v0.13.10, Windows 10 (10.0.19045.3930/22H2/2022Update)
AMD Ryzen 9 3900XT, 1 CPU, 24 logical and 12 physical cores
.NET SDK 8.0.100
[Host] : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX2
ShortRun : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX2

Job=ShortRun IterationCount=3 LaunchCount=1
WarmupCount=3

```
| Method | NumbersOfCalls | NumbersOfClasses | Mean | Error | StdDev | Ratio | RatioSD | Gen0 | Allocated | Alloc Ratio |
|------- |--------------- |----------------- |-------------:|------------:|-----------:|------:|--------:|-------:|----------:|------------:|
| **Jab** | **1** | **1** | **23.67 ns** | **19.60 ns** | **1.074 ns** | **1.00** | **0.00** | **0.0095** | **80 B** | **1.00** |
| MEDI | 1 | 1 | 101.19 ns | 73.78 ns | 4.044 ns | 4.28 | 0.02 | 0.0401 | 336 B | 4.20 |
| | | | | | | | | | | |
| **Jab** | **1** | **2** | **34.37 ns** | **21.20 ns** | **1.162 ns** | **1.00** | **0.00** | **0.0124** | **104 B** | **1.00** |
| MEDI | 1 | 2 | 152.44 ns | 48.25 ns | 2.645 ns | 4.44 | 0.08 | 0.0429 | 360 B | 3.46 |
| | | | | | | | | | | |
| **Jab** | **1** | **3** | **46.67 ns** | **19.28 ns** | **1.057 ns** | **1.00** | **0.00** | **0.0153** | **128 B** | **1.00** |
| MEDI | 1 | 3 | 217.12 ns | 207.52 ns | 11.375 ns | 4.66 | 0.35 | 0.0458 | 384 B | 3.00 |
| | | | | | | | | | | |
| **Jab** | **10** | **1** | **213.94 ns** | **34.73 ns** | **1.904 ns** | **1.00** | **0.00** | **0.0956** | **800 B** | **1.00** |
| MEDI | 10 | 1 | 836.62 ns | 30.44 ns | 1.669 ns | 3.91 | 0.04 | 0.4015 | 3360 B | 4.20 |
| | | | | | | | | | | |
| **Jab** | **10** | **2** | **327.55 ns** | **154.36 ns** | **8.461 ns** | **1.00** | **0.00** | **0.1240** | **1040 B** | **1.00** |
| MEDI | 10 | 2 | 1,429.73 ns | 1,090.76 ns | 59.788 ns | 4.36 | 0.12 | 0.4292 | 3600 B | 3.46 |
| | | | | | | | | | | |
| **Jab** | **10** | **3** | **471.03 ns** | **225.45 ns** | **12.358 ns** | **1.00** | **0.00** | **0.1526** | **1280 B** | **1.00** |
| MEDI | 10 | 3 | 2,067.34 ns | 244.00 ns | 13.374 ns | 4.39 | 0.14 | 0.4578 | 3840 B | 3.00 |
| | | | | | | | | | | |
| **Jab** | **100** | **1** | **2,087.62 ns** | **1,052.17 ns** | **57.673 ns** | **1.00** | **0.00** | **0.9537** | **8000 B** | **1.00** |
| MEDI | 100 | 1 | 8,277.90 ns | 214.87 ns | 11.778 ns | 3.97 | 0.11 | 4.0131 | 33600 B | 4.20 |
| | | | | | | | | | | |
| **Jab** | **100** | **2** | **3,275.71 ns** | **2,345.95 ns** | **128.590 ns** | **1.00** | **0.00** | **1.2398** | **10400 B** | **1.00** |
| MEDI | 100 | 2 | 13,711.81 ns | 3,013.26 ns | 165.167 ns | 4.19 | 0.17 | 4.3030 | 36000 B | 3.46 |
| | | | | | | | | | | |
| **Jab** | **100** | **3** | **4,289.60 ns** | **1,056.13 ns** | **57.890 ns** | **1.00** | **0.00** | **1.5259** | **12800 B** | **1.00** |
| MEDI | 100 | 3 | 21,340.73 ns | 657.08 ns | 36.017 ns | 4.98 | 0.07 | 4.5776 | 38400 B | 3.00 |
Loading