Skip to content

Commit 67b6b48

Browse files
davidkroelldavidkroell
committed
Add benchmark results in readme
Signed-off-by: David Kröll <[email protected]>
1 parent acf743a commit 67b6b48

File tree

2 files changed

+211
-0
lines changed

2 files changed

+211
-0
lines changed

‎README.md

+211
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,211 @@
1+
# shortcut
2+
3+
Shortcut is an URL Shortener written in Go. It's fully featured with a JSON API and an in-memory cache.
4+
5+
## Benchmark
6+
7+
For an URL Shortener it's very important to satisfy as most request per seconds as possible.
8+
Therefore, a benchmark is conducted.
9+
I am measuring the speed of shortcut with `wrk 4.1.0` in a VMWare Workstation 15 virtual machine.
10+
In order to find out what slows down the application, various approaches were pursued.
11+
12+
| | Host | Virtual Machine |
13+
| ----------- | ----------------------- | ----------------------------- |
14+
| OS | Windows 10 (1803) | Fedora 29 (Linux 4.20) |
15+
| CPU | Intel i7-7500 Dual Core | Intel i7-7500 Dual Core |
16+
| Memory | 16GB | 4GB |
17+
| Go version | | go 1.12 linux/amd64 |
18+
| Database | | MySQL 8.0.15 Community Server |
19+
20+
21+
#### Original
22+
The original application writes every request to the database (logging table). Furthermore, any request is logged on stdout.
23+
This code uses UUID as primary keys.
24+
25+
```bash
26+
$ wrk -c 256 -d 5s -t 48 http://localhost:9999/asdf
27+
Running 5s test @ http://localhost:9999/asdf
28+
48 threads and 256 connections
29+
Thread Stats Avg Stdev Max +/- Stdev
30+
Latency 244.27ms 44.09ms 364.56ms 81.33%
31+
Req/Sec 20.47 9.14 50.00 43.73%
32+
4827 requests in 5.10s, 735.36KB read
33+
Requests/sec: 946.24
34+
Transfer/sec: 144.15KB
35+
```
36+
37+
#### No DB writes
38+
After turning off the writes to the database, the performance is increased by around 3,5 times.
39+
```
40+
$ wrk -c 256 -d 5s -t 48 http://localhost:9999/asdf
41+
Running 5s test @ http://localhost:9999/asdf
42+
48 threads and 256 connections
43+
Thread Stats Avg Stdev Max +/- Stdev
44+
Latency 145.37ms 239.99ms 1.92s 89.23%
45+
Req/Sec 70.07 36.56 230.00 66.12%
46+
16859 requests in 5.10s, 2.51MB read
47+
Socket errors: connect 0, read 0, write 0, timeout 5
48+
Requests/sec: 3307.07
49+
Transfer/sec: 503.81KB
50+
```
51+
52+
#### No DB writes and no Stdout logging
53+
The below benchmark shows the performance when the logging to Stdout is disabled.
54+
In addition, no database writes took place.
55+
Shortcut can now serve more than 5500 requests per second, or 2000 more than with request logging.
56+
It's about 5 times as fast as the original version.
57+
```bash
58+
$ wrk -c 256 -d 5s -t 48 http://localhost:9999/asdf
59+
Running 5s test @ http://localhost:9999/asdf
60+
48 threads and 256 connections
61+
Thread Stats Avg Stdev Max +/- Stdev
62+
Latency 94.73ms 166.54ms 1.47s 89.81%
63+
Req/Sec 119.01 55.01 580.00 70.32%
64+
28520 requests in 5.10s, 4.24MB read
65+
Requests/sec: 5592.40
66+
Transfer/sec: 851.97KB
67+
```
68+
69+
#### Turn off additional middlewares
70+
There is an additional middleware in place, which sets the correct HTTP headers in POST requests.
71+
Disabling this does not really affect the performance.
72+
```bash
73+
$ wrk -c 256 -d 5s -t 48 http://localhost:9999/asdf
74+
Running 5s test @ http://localhost:9999/asdf
75+
48 threads and 256 connections
76+
Thread Stats Avg Stdev Max +/- Stdev
77+
Latency 93.29ms 174.71ms 1.74s 91.20%
78+
Req/Sec 117.97 49.61 290.00 68.20%
79+
28607 requests in 5.10s, 6.44MB read
80+
Requests/sec: 5609.12
81+
Transfer/sec: 1.26MB
82+
```
83+
84+
#### Integer primary keys
85+
The original version of shortcut uses UUID primary keys.
86+
The benchmark below shows the performance difference with INT auto increment keys.
87+
It's using ordinary 32bit keys (`INT`), which increases the performance by 15% (to original version).
88+
```bash
89+
$ wrk -c 256 -d 5s -t 48 http://localhost:9999/asdf
90+
Running 5s test @ http://localhost:9999/asdf
91+
48 threads and 256 connections
92+
Thread Stats Avg Stdev Max +/- Stdev
93+
Latency 213.82ms 33.56ms 344.21ms 85.51%
94+
Req/Sec 23.24 9.23 50.00 69.73%
95+
5604 requests in 5.10s, 853.73KB read
96+
Requests/sec: 1098.87
97+
Transfer/sec: 167.41KB
98+
```
99+
100+
#### Integer primary keys - no DB writes
101+
With 32bit integer keys and no writes to the database, the performance is merely the same as with UUID keys.
102+
This shows, that UUID keys take longer to write into the tables (more data).
103+
```bash
104+
$ wrk -c 256 -d 5s -t 48 http://localhost:9999/asdf
105+
Running 5s test @ http://localhost:9999/asdf
106+
48 threads and 256 connections
107+
Thread Stats Avg Stdev Max +/- Stdev
108+
Latency 142.05ms 226.85ms 1.71s 88.93%
109+
Req/Sec 70.76 36.25 240.00 65.88%
110+
16908 requests in 5.10s, 2.52MB read
111+
Socket errors: connect 0, read 0, write 0, timeout 6
112+
Requests/sec: 3315.60
113+
Transfer/sec: 505.11KB
114+
```
115+
116+
#### Big integer primary keys
117+
Utilizing 64bit integer (`BIGING`), the application is 7% faster than the original one.
118+
```bash
119+
$ wrk -c 256 -d 5s -t 48 http://localhost:9999/asdf
120+
Running 5s test @ http://localhost:9999/asdf
121+
48 threads and 256 connections
122+
Thread Stats Avg Stdev Max +/- Stdev
123+
Latency 230.42ms 45.04ms 355.42ms 78.07%
124+
Req/Sec 21.76 10.17 70.00 58.99%
125+
5171 requests in 5.10s, 787.77KB read
126+
Requests/sec: 1014.14
127+
Transfer/sec: 154.50KB
128+
```
129+
130+
#### Big integer primary keys - no DB writes
131+
Without writing the logs to the database, the performance is again just the same.
132+
```bash
133+
$ wrk -c 256 -d 5s -t 48 http://localhost:9999/asdf
134+
Running 5s test @ http://localhost:9999/asdf
135+
48 threads and 256 connections
136+
Thread Stats Avg Stdev Max +/- Stdev
137+
Latency 132.12ms 223.68ms 1.89s 90.54%
138+
Req/Sec 69.41 34.58 212.00 68.95%
139+
16715 requests in 5.10s, 2.49MB read
140+
Socket errors: connect 0, read 0, write 0, timeout 21
141+
Requests/sec: 3278.92
142+
Transfer/sec: 499.52KB
143+
```
144+
145+
#### Big integer primary keys - no DB writes and no Stdout logging
146+
Disabling the request logger changes nothing. This again shows,
147+
that only the writes to the database slow down the application.
148+
```bash
149+
$ wrk -c 256 -d 5s -t 48 http://localhost:9999/asdf
150+
Running 5s test @ http://localhost:9999/asdf
151+
48 threads and 256 connections
152+
Thread Stats Avg Stdev Max +/- Stdev
153+
Latency 81.29ms 132.48ms 1.22s 89.46%
154+
Req/Sec 124.34 52.48 340.00 66.36%
155+
29950 requests in 5.10s, 4.46MB read
156+
Requests/sec: 5873.59
157+
Transfer/sec: 0.87MB
158+
```
159+
160+
### Caching
161+
In order to further increase the performance, I've written an in-memory cache,
162+
which works like the following diagram outlines.
163+
164+
![cache_flow](Cache flow diagram)
165+
166+
#### Cache - naive approach
167+
The below benchmark shows the performance with just the cache enabled.
168+
The database writes and writes to Stdout are enabled, therefore, does not affect the performance.
169+
```bash
170+
$ wrk -c 256 -d 5s -t 48 http://localhost:9999/asdf
171+
Running 5s test @ http://localhost:9999/asdf
172+
48 threads and 256 connections
173+
Thread Stats Avg Stdev Max +/- Stdev
174+
Latency 276.82ms 57.93ms 360.83ms 80.76%
175+
Req/Sec 19.20 9.99 111.00 78.90%
176+
4282 requests in 5.10s, 652.34KB read
177+
Requests/sec: 839.43
178+
Transfer/sec: 127.88KB
179+
```
180+
181+
#### Cache - no DB writes
182+
The following benchmark shows the performance without database writes.
183+
Request logging to Stdout is enabled. It's 13 (!) times faster than the original version.
184+
```bash
185+
$ wrk -c 256 -d 5s -t 48 http://localhost:9999/asdf
186+
Running 5s test @ http://localhost:9999/asdf
187+
48 threads and 256 connections
188+
Thread Stats Avg Stdev Max +/- Stdev
189+
Latency 20.87ms 16.88ms 171.51ms 75.25%
190+
Req/Sec 264.09 111.06 1.43k 83.04%
191+
63561 requests in 5.10s, 9.46MB read
192+
Requests/sec: 12465.42
193+
Transfer/sec: 1.85MB
194+
```
195+
196+
#### Cache - no DB writes and no Stdout logging
197+
The below benchmark shows the performance when request logging is disabled.
198+
Writes to the database are disabled too.
199+
This increases the performance by 2,7 to the version above.
200+
In addition, it is 35 (!) times faster than the original one.
201+
```bash
202+
$ wrk -c 256 -d 5s -t 48 http://localhost:9999/asdf
203+
Running 5s test @ http://localhost:9999/asdf
204+
48 threads and 256 connections
205+
Thread Stats Avg Stdev Max +/- Stdev
206+
Latency 127.03ms 227.65ms 1.61s 86.49%
207+
Req/Sec 0.90k 1.72k 11.60k 86.13%
208+
173109 requests in 5.10s, 38.96MB read
209+
Requests/sec: 33949.68
210+
Transfer/sec: 7.64MB
211+
```

‎cache_flow.jpg

40.3 KB
Loading

0 commit comments

Comments
 (0)