You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# Benchmarking in Swift with `swift-collections-benchmark`
11
10
12
-
There is an age-old adage of programming which states *"Make it work, then make it right, then make it fast."* Today we will be focusing on how to *make it fast*, with the help of a valuable tool called Benchmarking.
11
+
There is an age-old adage of programming which states *"Make it work, then make it right, then make it fast."* Today we will be focusing on how to *make it fast*, with the help of a valuable technique called Benchmarking.
13
12
14
13
When developing software, especially when working with algorithms and data structures, performance is often a key concern. You may have experienced a piece of code that behaves well in your tests, but when it's exposed to real-world data, performance degrades. This is where **benchmarking** comes in. Benchmarking allows developers to measure how long a piece of code takes to run, helping to identify bottlenecks and areas for optimization.
15
14
@@ -25,7 +24,7 @@ Benchmarking is somewhat analogous to **snapshot testing**, but for performance.
25
24
26
25
### Comparison to Other Testing Types
27
26
28
-
-**Unit Testing**: This checks if a small, isolated piece of code produces the correct result. It’s focused on correctness rather than performance.
27
+
-**Unit Testing**: This checks if a small, isolated piece of code produces the correct result. It’s focused on correct behavior rather than performance.
29
28
-**Snapshot Testing**: This captures a "snapshot" of how a piece of code (often UI code) looks or behaves at a point in time, and tests future runs against that snapshot to detect changes. Again, it’s about correctness but not performance.
30
29
-**Performance Testing**: This is about analyzing the exact speed or memory usage.
31
30
-**Benchmark Testing**:
@@ -35,9 +34,9 @@ Benchmarking is somewhat analogous to **snapshot testing**, but for performance.
35
34
[^2]: Benchmark Testing is actually a subset of Performance Testing, not a separate category.
36
35
37
36
38
-
|Testing Type| Single Test (Focus on Individual Case) | Comparison Against Prior Runs (Regression) |
37
+
|What We're Testing| Single Test (Focus on Individual Case) | Comparison Against Prior Runs (Regression) |
@@ -47,41 +46,17 @@ Benchmarking is somewhat analogous to **snapshot testing**, but for performance.
47
46
48
47
## Benchmarking with `swift-collections-benchmark`
49
48
50
-
In 2021, Swift.org announced [Swift Collections](https://www.swift.org/blog/swift-collections/) an open-sourced package with more advanced data structures than those provided by the standard library. In order to develop Swift Collections, they developed [`swift-collections-benchmark`](https://github.com/apple/swift-collections-benchmark). This package is a great tool for benchmarking in Swift, particularly for comparing collections and algorithms. It provides a flexible framework for running performance tests and collecting detailed data on how different implementations behave under various conditions.
49
+
In 2021, Swift.org announced [Swift Collections](https://www.swift.org/blog/swift-collections/), an open-sourced package with more advanced data structures than those provided by the standard library. In order to develop Swift Collections, they developed [swift-collections-benchmark](https://github.com/apple/swift-collections-benchmark). This package is a great tool for benchmarking in Swift, particularly for comparing collections and algorithms. It provides a flexible framework for running performance tests and collecting detailed data on how different implementations behave under various conditions.
51
50
52
51
To show how this works, let’s dive into a simple example comparing two common collection types in Swift: `Array` and `Set`. Remember that an `Array` is an ordered `Collection` of values. A `Set` is similar to an `Array` but with two major differences: it is unordered, and it cannot contain duplicate values. Let's find out which has better performance...
53
52
54
53
## Example: Array vs Set Performance
55
54
56
55
### Problem Setup
57
56
58
-
Suppose we want to compare how long it takes to check whether a collection contains a specific element. We'll compare the performance of an `Array` and a `Set` when performing this lookup.
57
+
Suppose we want to decide if we should use an `Array` or a `Set`. We're currently using an `Array` but we suspect a `Set might be faster. Let's test the performance of both across our use cases to see which is faster.
59
58
60
-
### Step 1: Create anExample with an Array
61
-
62
-
```swift
63
-
let array =Array(1...1000000)
64
-
65
-
funccontainsInArray(_value: Int) ->Bool {
66
-
return array.contains(value)
67
-
}
68
-
```
69
-
70
-
In this example, we’re creating an array with 1,000,000 elements, and a simple function `containsInArray` that checks whether a given value is present in the array.
71
-
72
-
### Step 2: Create an Example with a Set
73
-
74
-
```swift
75
-
letset=Set(1...1000000)
76
-
77
-
funccontainsInSet(_value: Int) ->Bool {
78
-
returnset.contains(value)
79
-
}
80
-
```
81
-
82
-
Similarly, we create a `Set` with 1,000,000 elements, and a function `containsInSet` that checks for the presence of a value.
83
-
84
-
### Step 3: Benchmarking the Performance
59
+
### Benchmarking the Performance
85
60
86
61
To benchmark these two implementations, we’ll use the `swift-collections-benchmark` package to compare how long each function takes to execute.
87
62
@@ -95,87 +70,130 @@ dependencies: [
95
70
]
96
71
```
97
72
98
-
Then, import the package:
99
-
73
+
Next, lets add the dependency to our target in our Package.swift file:
Here’s how we can use the `swift-collections-benchmark` framework to compare the performance of `containsInArray` and `containsInSet`:
107
85
108
86
```swift
109
87
importCollectionsBenchmark
110
88
111
-
let benchmark =Benchmark(title: "Array vs Set Contains Performance")
89
+
// Create a benchmark test suite
90
+
var benchmark =Benchmark(title: "ArrayVsSet Benchmark")
112
91
113
-
// Add the array test
92
+
// Add your tests here
114
93
benchmark.addSimple(
115
-
title: "Array contains",
116
-
input: Int.self
117
-
) { _in
118
-
_=containsInArray(999999)
94
+
title: "Array<Int> init",
95
+
input: Int.self// 👈🏼 input type
96
+
) { inputin
97
+
blackHole(Array(0..<input))
119
98
}
120
99
121
-
// Add the set test
122
100
benchmark.addSimple(
123
-
title: "Set contains",
101
+
title: "Set<Int> init",
124
102
input: Int.self
125
-
) { _in
126
-
_=containsInSet(999999)
103
+
) { inputin
104
+
blackHole(Set(0..<input)) // 👈🏼 What's blackhole? Read on and find out.
127
105
}
128
106
129
-
// Run the benchmark
130
-
benchmark.run()
107
+
// Run your benchmark tests
108
+
benchmark.main()
131
109
```
132
110
133
-
In this benchmark, we add two tests—one for checking if an array contains the value `999,999`, and another for checking if a set contains the same value. Finally, we run the benchmark to see the results.
134
-
135
-
### Step 5: Run the Benchmark Test
111
+
In this benchmark, we add two tests, one for checking how long it takes to initialize an Array, and one for a Set. Notice how tell the framework the type of input we'll put into this test. CollectionsBenchmark will create randomized input to try running the same test repeatedly at different sizes. Types like `Int.self` and `[Int].self` are already built into the framework. If you want to use a custom type, you'll need to register a custom generator using the `registerInputGenerator` method on `Benchmark`.
136
112
137
-
Now you can run the tests directly from the command line. It's important to run the tests in **release mode** to ensure you're getting accurate performance measurements, as Swift applies optimizations in this mode that significantly affect performance.
113
+
### Run the Benchmark Test
138
114
115
+
Remember that our target is an `executableTarget`. This is because CollectionsBenchmark is designed to be run from the command line. Here's an example of how to run it:
139
116
140
-
Run the benchmark from the command line using the following command:
141
-
142
-
```bash
143
-
swift run -c release <benchmark-target>
117
+
```zsh
118
+
swift run -c release ArrayVsSetBenchmark run --cycles 3 results
144
119
```
145
-
-`swift run` runs your project.
146
-
-`-c release` ensures the benchmark is executed in **release mode** for accurate results.
147
-
-`<benchmark-target>` refers to the target containing your benchmark code.
148
120
149
-
For example:
150
-
```bash
151
-
swift run -c release MyBenchmarkTarget
152
-
```
121
+
Let's break it down:
122
+
-`swift`: We'll use the Swift CLI to build and run the target.
123
+
-`run`: We will run the target (which requires building)
124
+
-`-c release`: It is important to run benchmark tests in a release build. Why? See below...
125
+
-`ArrayVsSetBenchmark`: the name of the executable target as defined in our SPM package
126
+
-`run`: This second `run` is telling BenchmarkCollections to run the benchmark. To see other commands, try using `help`.
127
+
-`--cycles 3`: We will rerun the benchmark test 3 times to get more data, and calculate averages.
128
+
-`results`: Output the benchmark results into a JSON file named `results` in the same directory.
153
129
154
130
The command will output detailed results, including execution time, standard deviation, and number of iterations for each test case.
155
131
156
132
For more detailed documentation on setting up and running benchmarks, visit the official [`swift-collections-benchmark` documentation](https://github.com/apple/swift-collections-benchmark/blob/main/Documentation/01%20Getting%20Started.md).
157
133
158
134
#### Why It’s Important to Benchmark in Release Configuration
159
135
160
-
Benchmarking in **release** configuration is crucial because Swift optimizes code differently in debug and release configurations. In **debug** configuration, the compiler prioritizes features like code readability and easier debugging, which results in slower execution due to the absence of key optimizations. However, in **release** configuration, the compiler applies aggressive optimizations such as inlining, dead code elimination, and loop unrolling to improve performance.
136
+
Benchmarking in **release** configuration is crucial because Swift optimizes code differently in debug and release configurations. In **debug** configuration, the compiler prioritizes features like code readability and easier debugging, which results in slower execution due to the absence of key optimizations. However, in **release** configuration, the compiler applies aggressive optimizations such as inlining, dead code elimination, and loop unrolling to improve performance. As a general rule **debug** configuration leads to faster compilation, but slower runtime, while **release** has slower compilation and faster runtime. To accurately measure how your code will behave in a real-world environment, always benchmark in release mode to ensure you're evaluating the fully optimized version of your code.
137
+
138
+
That being said, if your benchmark fails to run, you may need to run it in **debug** so that you can fix the problem. BenchmarkCollections will provide more helpful information if you run in **debug**.
161
139
162
-
If you benchmark in debug, you may end up with misleading results that don’t reflect the true performance of your code in production. To accurately measure how your code will behave in a real-world environment, always benchmark in release mode to ensure you're evaluating the fully optimized version of your code.
140
+
### What is `blackhole()`
141
+
You might have noticed a strange blackhole in our tests above:
163
142
164
-
### Step 6: Analyze the Results
143
+
```swift
144
+
blackHole(Set(0..<input))
145
+
```
146
+
147
+
In swift-collections-benchmark, a black hole is used to prevent the Swift compiler's optimizer from removing what it might consider "unused" code during benchmarks. When measuring performance, if the optimizer detects that the results of an operation aren't being used, it might skip those operations entirely, leading to inaccurate benchmarks. A black hole function consumes the result of a computation in such a way that it can't be optimized away, ensuring the benchmark measures the actual execution time of the operations being tested. This guarantees the integrity and accuracy of performance measurements.
148
+
149
+
### Analyze the Results
165
150
166
151
When you run the benchmark, you should see output like the following:
In this case, you’ll likely notice that the `Set` performs significantly better than the `Array` for lookups. This is because `Set` is implemented as a hash table, which provides constant-time complexity (`O(1)`) for lookups, whereas `Array` performs a linear search (`O(n)`).
173
+
Just like the console says, there should be a new file named `results` with the results of the benchmark tests. If we open the file, we can see that it's a JSON file. This file will also be persisted across tests, so that you can compare current results to past results. This empowers you to catch performance regressions. Now, let's render these results into a format that is more useful.
174
+
175
+
```zsh
176
+
swift run -c release ArrayVsSetBenchmark render results chart.png
177
+
```
178
+
179
+
Using the `render` command from `CollectionsBenchmark` we now have a new `chart.png` file that can visually show us our results. It should look something like this:
180
+
181
+
182
+
183
+
On the x axis we can see the count of how many operations were performed by the framework. They increase exponentially as you move to the right. On the y axis we see the amount of time it took for those operations to be performed. They also increase exponentially as you move from the bottom to the top. Great! So what does all this mean?
184
+
185
+
As you can see the performance characteristics of a `Set` and an `Array` are extremely similar. In fact, they are so similar that you probably don't need to care about the performance differences between them. The majority of the lines on the graph are roughly flat. With this scale, that means that they are operating at an O(n) complexity.
186
+
187
+
But there are two very big exceptions: "Array<Int> removeFirst" and "Array<Int> insert at 0". Both of these tests have a noticeably steeper slope. Given the steepness of the slope it appears that they are running at O(n) time complexity. In other words, the increase in time is directly proportional to the amount of items in the operation. If we look at the documentation for these two methods, we will see that they are in fact expected to run at O(n) time complexity.[^3]
188
+
189
+
[^3]: Strangely, it seems that `array.insert(num, at: 0)` actually performs faster than the other insertion methods until we reach about 2,000 items. Then it performs exponentially slower.
190
+
191
+
From our analysis, we should be able to learn this: An `Array` and a `Set` have remarkably similar performance except for some key use cases. If your use cases is one of those use cases, and particularly if you are dealing with large data sets, you should perhaps consider switching from an `Array` to a `Set`.
176
192
177
193
## Conclusion
178
194
179
195
Benchmarking is an essential tool for ensuring that your code not only works, but works efficiently. While unit tests ensure correctness, benchmarking ensures performance remains consistent as your code evolves. Using the `swift-collections-benchmark` package makes it easy to measure and compare the performance of different implementations, as we demonstrated with the `Array` vs `Set` example.
180
196
181
-
By adding benchmark tests to your development workflow, you can catch potential performance regressions early and make data-driven decisions about how to optimize your code.
197
+
By adding benchmark tests to your development workflow, you can catch potential performance regressions early and make data-driven decisions about how to optimize your code. If you would like to see the rest of the code I used to make this article to see [ArrayVsSet](https://github.com/DandyLyons/ArrayVsSet) on GitHub.
198
+
199
+
Have any questions? See any mistakes or areas that I could improve this article? Please message me on [Mastodon](https://iosdev.space/@dandylyons).
0 commit comments