build: use profile-guided optimization

[nvanbenschoten]

Go 1.20 (https://tip.golang.org/doc/go1.20) added support for profile-guided optimization (PGO). Go 1.21 (https://tip.golang.org/doc/go1.21) further improves PGO and marks it as ready for general use.

PGO promises to improve the performance of Go programs:

Performance improvements vary depending on application behavior, with most programs from a representative set of Go programs seeing between 2 and 7% improvement from enabling PGO.

PGO currently achieves this by inline hot function calls, but the Go team expects it to become more capable in future releases: golang/go#62463.

Once #109773 lands, we should test out PGO and explore integrating it into our CI pipeline.

Jira issue: CRDB-31351

Epic CRDB-41952

[nvanbenschoten]

Go 1.22 is expanding PGO's capabilities:

Profile-guided Optimization (PGO) builds can now devirtualize a higher proportion of calls than previously possible. Most programs from a representative set of Go programs now see between 2 and 14% improvement from enabling PGO.

[lyang24]

I read more about pgo its very interesting - the hard part is it needs a pprof to guide the optimization, do you have guidances on how get a proper profile?

bazel already support pgo, once i get profile i can drop into the folder that contains the main.go, and add pgoprofile = ":<profile_name>.pprof", to
https://github.com/lyang24/cockroach/blob/d0bb7e47c6649dce2d8efa0c26b999ed5d161fe6/pkg/cmd/cockroach/BUILD.bazel#L25-L30

[rickystewart]

Hi @lyang24, in release-24.1 you can manually collect a profile on the Advanced Debug page or if you wish there is a method to set up automatic profiling. Here is a draft commit showing how to add profiles to the build (the profiles here are one-offs for an internal experiment that we did). You could also directly use the profiles from this commit if you like. If you do please be aware that we do not intend to merge this commit as-is and the provided profiles may not be representative and the PGO build may not be "production-ready"; your mileage may vary and this is not supported by Cockroach Labs.

Note that building with PGO is substantially slower than building without it. It also has implications for caching with respect to the build especially if you are doing development, running tests, etc.

[tbg]

When this gets tackled we'll also want to include a sysbench profile. (I'm happy to provide one, just DM me).

[tbg]

Some updates since none of this pinged back to this issue (I think it was all tracked under https://cockroachlabs.atlassian.net/browse/CRDB-44692, which is not linked to Github):

There have been numerous PRs for collecting profiles from our nightly roachtest runs:
#137363
#137365
#136919
#136318

pgo support for bazel (i.e. ability to plug in cpu profiles for pgo in release builds) was added here but then reverted due to OOM issues on the builder machines:

#137715
#137729
#137863

[tbg]

I tested on this build shared by @rickystewart . The profile used for pgo is from the following roachtests: kv0/enc=false/nodes=3,kv95/enc=false/nodes=3,tpccbench/nodes=3/cpu=16

The results were that there wasn't a discernable change on the sysbench reference workload. This is surprising, so we should evaluate it again, perhaps on a pgo build based solely on a profile from oltp_read_write.

$ benchstat def.txt pgo.txt
           │   def.txt   │             pgo.txt              │
           │ queries/sec │ queries/sec  vs base             │
Sysbench/X   26.39k ± 2%   26.58k ± 2%  ~ (p=0.305 n=20+16)

           │   def.txt   │             pgo.txt              │
           │  txns/sec   │  txns/sec    vs base             │
Sysbench/X   1.320k ± 2%   1.329k ± 2%  ~ (p=0.305 n=20+16)

           │  def.txt   │             pgo.txt             │
           │   ms/min   │   ms/min    vs base             │
Sysbench/X   11.93 ± 1%   12.00 ± 3%  ~ (p=0.665 n=20+16)

           │  def.txt   │             pgo.txt             │
           │   ms/avg   │   ms/avg    vs base             │
Sysbench/X   48.50 ± 3%   48.15 ± 1%  ~ (p=0.305 n=20+16)

           │  def.txt   │             pgo.txt             │
           │   ms/p95   │   ms/p95    vs base             │
Sysbench/X   75.82 ± 2%   75.14 ± 3%  ~ (p=0.302 n=20+16)

           │  def.txt   │             pgo.txt             │
           │   ms/max   │   ms/max    vs base             │
Sysbench/X   174.4 ± 5%   172.9 ± 7%  ~ (p=0.560 n=20+16)

[rickystewart]

We updated the profile-generation logic as well as the profile itself. Sysbench results using this profile show reasonable improvements in the 1-2% range:

cockroach$ ../../../../bin/benchstat before.txt after.txt                                  
                                                 │ before.txt  │             after.txt              │
                                                 │ queries/sec │ queries/sec  vs base               │
Sysbench/a=oltp_read_write/nodes=3/cpu=8/conc=64   22.58k ± 1%   22.99k ± 1%  +1.82% (p=0.013 n=20)

                                                 │ before.txt  │             after.txt              │
                                                 │  txns/sec   │  txns/sec    vs base               │
Sysbench/a=oltp_read_write/nodes=3/cpu=8/conc=64   1.129k ± 1%   1.149k ± 1%  +1.82% (p=0.013 n=20)

                                                 │ before.txt │             after.txt             │
                                                 │   ms/min   │   ms/min    vs base               │
Sysbench/a=oltp_read_write/nodes=3/cpu=8/conc=64   13.25 ± 2%   12.90 ± 2%  -2.64% (p=0.045 n=20)

                                                 │ before.txt │             after.txt             │
                                                 │   ms/avg   │   ms/avg    vs base               │
Sysbench/a=oltp_read_write/nodes=3/cpu=8/conc=64   56.70 ± 1%   55.69 ± 1%  -1.78% (p=0.013 n=20)

                                                 │ before.txt │          after.txt           │
                                                 │   ms/p95   │   ms/p95    vs base          │
Sysbench/a=oltp_read_write/nodes=3/cpu=8/conc=64   89.97 ± 3%   88.36 ± 3%  ~ (p=0.083 n=20)

                                                 │ before.txt  │           after.txt           │
                                                 │   ms/max    │   ms/max     vs base          │
Sysbench/a=oltp_read_write/nodes=3/cpu=8/conc=64   224.7 ± 11%   220.9 ± 20%  ~ (p=0.968 n=20)
co

This is not as big of an improvement as we would have hoped, but we can make improvements to the profile over time. That will be done as follow-up work. We also expect the Go compiler to get "smarter" about applying PGO-specific optimizations with new releases.