Use custom profiles to set LTO and such

[matklad]

Rust 1.57 stabilized custom profiles feature of cargo:

rust-lang/cargo#9943

That means that we can switch from using env variables for LTO

nearcore/Makefile

Lines 1 to 2 in d1d4559

	export CARGO_PROFILE_RELEASE_CODEGEN_UNITS = 1
	export CARGO_PROFILE_RELEASE_LTO = fat

to using a proper --profile production

Steps here:

• Add custom profile to root Cargo.toml
• Search all the places that currently set CARGO_PROFILE_RELEASE_LTO (makefile, estimator, maybe something else?)
• Update them all to pass --profile production instead.

[matklad]

cc @jakmeier for estimator changes.

[mina86]

Turns out this isn’t as trivial as I thought at first. Issue is that with --profile production the binary ends up at target/production/neard which means that places which refer to target/release would need to be updated as well.

I wonder that maybe it would be better to enable LTO in release profile and introduce dev-release profile equivalent to current release? Just thinking out loud.

[matklad]

Good point. I'd probably still lean towards for introduction of prod profile: today, to build production artifacts, you are going to use a custom, unusual workflow anyway (make), so adding more stuff there doesn't affect conceptual complexity (it's still a custom command!). In contrast, for workflows like "running a test in release mode to access performance" the standard cargo test --release works, and making that into non-standard cargo test --profile dev-release does add to conceptual complexity.

Ideally, dev-release would be a standard Cargo profile but, alas, it isn't

[abacabadabacaba]

I think that release profile should be what we are actually releasing. Anything less is not a release, it's "debug with optimizations" or something like that.

[mina86]

I'd probably still lean towards for introduction of prod profile: today, to build production artifacts, you are going to use a custom, unusual workflow anyway (make), so adding more stuff there doesn't affect conceptual complexity (it's still a custom command!).

The issue is that people expect make release to put all the files in target/release which will no longer be true with prod profile. I’d rather avoid a situation where someone was testing some code with cargo build --release and later made a release by running make release and copying files from target/release. We can copy the files over in Makefile but then we just made the build commands more complicated for no discernable reason. As you know, there is --out-dir but it’s unstable so we probably don’t want to use it.

I think that release profile should be what we are actually releasing.

The issue with that is that what we’re releasing takes ages and heaps of memory to build. This is why we disabled LTO in the first place.

And yes, I am aware that I’m the person who removed LTO from release in the first place and now seemingly suggest putting it back in. It’s more than I’m wondering what’s the best option now with custom profiles stabilised.

[matklad]

The issue with that is that what we’re releasing takes ages and heaps of memory to build. This is why we disabled LTO in the first place.

Ahh, so our Makefile doesn't do "link files from target/release to output directory". Or, rather, we only do this for sandbox in a rather unprincipled way. Yeah, then it's probably better to not change that just because. I am now 0.7 convinced that it's better to introduce dev-release and to mention it in the contributing.md.

[abacabadabacaba]

The issue with that is that what we’re releasing takes ages and heaps of memory to build.

For me, it takes less than 10 minutes to link neard on a GCP instance with 16 GiB RAM when using make release. So I think that this claim is kinda exaggerated.

[mina86]

For me, it takes less than 10 minutes to link neard on a GCP instance with 16 GiB RAM when using make release. So I think that this claim is kinda exaggerated.

My machine used to OOM when trying to `cargo build --release’. The measurement I’ve made back then show LTO build 70% slower with full build and 1400% slower with incremental build:

• Elapsed time of full builds:

  |          |   -j6 |  -j8 | -j12 | -j24 |
  | lto=off  |  5:16 |      | 3:22 | 2:58 |
  | lto=thin | 11:07 |      | 6:24 | 5:21 |
  | lto=fat  |  8:57 | 7:18 | OOM  | OOM  |
  

• Elapsed time of incremental builds:

  |          |  -j6 | -j12 | -j24 |
  | lto=off  | 0:10 | 0:10 | 0:10 |
  | lto=thin | 2:32 | 1:22 | 1:09 |
  | lto=fat  | 2:43 | 2:41 | OOM  |
  

Things seems to have improved since my machine no longer OOMs (though it does stagger a little when doing ‘make release’) but the issue is real. 10 minutes is a long time especially if you’d like to git bisect. So the question is more whether people do those kinds of things.

[mina86]

So what’s the consensus here? The three things we can choose from seem to be:

1. Leave things as they are now.

2. Use a custom profile for release build. This will remove a few export lines from Makefile but we will have to deal with executable ending up in a different directory. This perhaps wouldn’t be such a big issue except it seems we’ve got validators used to building things through make release and copying files from target/release. So we probably would need to add copying to Makefile but that sounds much more complicated than having a few export statements.

3. Enable LTO in release profile. This has the issue of increasing the time needed to cargo build --release which might be annoying during development. While for clean builds the increase in time might not be proportionally that big but for incremental builds it can get from several seconds to couple of minutes. Historically, LTO builds also resulted in OOMs and failed builds.

I’m leaning towards the first option. I don’t really see that much benefit of having the custom profile as at the end of the day it doesn’t clean things that much and I also don’t think that differences in LTO vs. non-LTO builds are so big that it would be significant during development.

[mina86]

OK, I’m gonna close this with no action taken. If anyone has arguments for doing 2 or 3 from my previous comment feel free to reopen.