Build script: Adding directive cargo::no-new-rust-code to speed up compilation #16062
Description
Activity
This seems similar in spirit to pipelined builds we do for Rust code where we allow downstream dependencies to build once the rmeta is created, only requiring the rlib at link time.
I suspect the first step in doing this would be to redefine build scripts as the directives being processed on the fly, rather than at build script completion. Support for doing so was a consideration in the discussions for #13233.
Unsure what all it would take architecturally in Cargo to have build script execution participate in the pipelined builds setup.
Potential routes forward:
- We consider all of the ins and outs of this and make a decision and move forward
- We identify small pieces we'd be willing to accept independent of this (e.g. processing build scripts on the fly) and then work towards a prototype to see what we think
Unsure which route we should go and how much we'd be willing to accept as part of this.
Not familiar with how cargo internally works, it seems to spawn one thread per rustc/build-script call to track the tasks?
In cc-rs we used to do that, but now we switched to using async to do spawning waiting for children, so that we don't have to spawn two threads for each compiler invocation:
And use a super dumb async reactor becaue we don't want to pull in extra crate or maintain it
Since cargo doesn't have the constraint cc-rs have, maybe it can just go full async, and use async to keep track of states?
I could make use of that in sys crates that run bindgen + compile fallback from source.
Splitting of build across multiple scripts doesn't seem helpful to me.
The problem is that bindgen must be run on the right version of the header files, which first requires detecting whether the sys library has a pre-installed version or needs to use a vendored one. If I had to split the steps into two build scripts, I would have to do this detection in each build script, duplicating the work (with a risk of two halves disagreeing if the detection goes out of sync for some reason). If there was some dependency graph and communication between multiple split build scripts I could make detection a one build script and then bindgen and compile from sources its children, but that starts looking overly complicated, and linking and potential overzealous antivirus checks of three build script executables will probably make everything slower.
BTW: this directive probably also means no new
cargo::metadatadirectives can be used, since the env vars they create are visible to Rust code in other deps.Reacted by Jiahao XUSemi-related rambling about possible usage of RFC 3632 "externally implementable functions" in build scripts
Maybe we could introduce "Build Script V2 ™️ " using something like rust-lang/rfcs#3632 , and have different "stages" of the build script as different functions the user can implement, e.g. something like
impl fn build_script_v2::bikeshed_before_rust_build() { // stuff that must complete before `rustc` can begin building this crate's source, e.g. code generation // or emitting directives that change cfgs } impl fn build_script_v2::bikeshed_concurrent_with_rust_build() { // stuff that can happen concurrent with `rustc` producing `.rmeta`, but before linking, // like running external build systems for `-sys` crates } // etc if there are other stages; all have a default body that does nothing
A "Build Script V2 ™️ " build script would be a library crate-type (not a binary itself) and would be compiled in as a dependency of a simple binary (source distributed with cargo and compiled on-the-fly) that calls the functions in the relevant order, telling cargo when each one has completed. If a directive is emitted in the "wrong" stage (i.e. after it is allowed), cargo would detect the error and fail compilation of the crate.
Also, perhaps some stages could be only executed during a
cargo build, not during acargo check.Maybe we could introduce "Build Script V2 ™️ " using something like rust-lang/rfcs#3632
That's interesting idea, if cargo build scripts is redesigned today then it'd surely be like this, plus combined with declarative build scripts, though I think it's orthogonal to this proposal, as it is a more radical and fundamental change?
I personally want to pursue a more incremental change that can be applied to existing build scripts relatively easily.
You can open a separate issue for that.
In today's Cargo meeting, we did talk about one component of this that we'd like to see in advance, as a separate step: reworking cargo to process build script directives as they arrive. That would have some advantages already, such as printing warnings/errors right away (e.g. if the script prints a warning before a long build process).
Apart from that, 👍 for @epage's point that this will need some additional design. There are a few potential designs that might make sense here. (Will add a separate comment on potential designs.)
Another potential alternative design:
Rather than adding a mechanism for the build script to ask cargo to pipeline it with the rest of the crate, we could add a way for a crate to consist only of a build script and no Rust code, and then when cargo builds that crate, it doesn't need to expect or wait on any Rust metadata for that crate, only a library to link against. Then, a crate building a library in its build script could split that build script into a separate crate, and those crates would be able to build in parallel using something like cargo's existing pipelining, without needing new additional complex logic to handle pipelining a build script relative to the rest of the crate containing that build script.
In response to the Cargo team discussion, I'm going to mark this
S-needs-designthough- The streaming build script logic is
S-accepted - This overall effort is also blocked on
link-only/no-linkbeing in a better condition. It doesn't need to be stabilized but at least working well enough for us to prototype on top of it. Last time someone tried to integrate it into Cargo, it wasn't working.
We felt that some upfront design work is needed for this because of how invasive a change this would be.
- Exploring alternative solutions like Build script: Adding directive cargo::no-new-rust-code to speed up compilation #16062 (comment)
- With the proposed solution, there is the question of how much we should "trust but verify". The proposal mentions new directives being an error and to not worry if rust code in
OUT_DIRis modified. Are we comfortable enough with ignoring extraOUT_DIRchanges?
- The streaming build script logic is
Thank you!
The alternative design raised by @joshtriplett have an issue:
It would complicate things for crates like zstd-sys, which might do bindgen on headers as well as compiling code, which is also echoed by @kornelski in this comment
Another alternative design would be to have multiple build scripts or multiple stages, the multi-stage design is raised by @zachs18 in this comment
The multiple build script, IIRC is also proposed by @joshtriplett but rather for declarative build-script?
For multi-stage it'd have functions run before rustc and functions that can run after, it would be radical change though.
For multiple-build scripts, we can do the same thing and declare some to be run before and some after.
Since multi build-scripts is already proposed and maybe start implementing, and it's still run as a binary not a shared lib, it's probably easier as long as the output directory is shared or someway to pass information down.
(I wonder how multiple build scripts are compiled? Would be more efficient to generate one multi-call binary)
Multi build-scripts remind me of cargo-script, which can have different dependencies for different build scripts, which might be helpful for multi build-script to reduce dependencies for each stage?
With the proposed solution, there is the question of how much we should "trust but verify". The proposal mentions new directives being an error and to not worry if rust code in OUT_DIR is modified. Are we comfortable enough with ignoring extra OUT_DIR changes?
I think the best/robust solution would be to copy entire OUT_DIR to a separate directory, then after compilation compare them using checksum
Some form of link-only crates or scripts could be useful for
cargo checkandcargo doctoo (and tools based on rustdoc JSON).Currently docs needlessly build C dependencies that won't be used. For docs.rs it's possible to work around it by checking for an env var, but that isn't a good solution for usual dev workflows, since toggling it rebuilds all crates downstream from the sys crate, rather than just the missing library.
However, a literal link-only dependency in Cargo isn't sufficient for build scripts that conditionally either detect or build a C library. The order of operations is wrong — the link only build needs to be spawned after the detection build script runs.
Maybe there could be a type of a build script that runs early, before a dependency build, and tells Cargo whether to proceed or skip it?
(Have I just rediscovered
./configure?)(Have I just rediscovered ./configure?)
Maybe we should go down the path, and split build-script into multiple based on that and run in order:
- Detect env and generate code
- Find external lib or compile vendored version, which will be automatically skipped for cargo-check/cargo-doc
Problem
Currently, all crates that bindgens, compiles or finds external dependencies have to wait for compilation/lookup of compiler artifacts to be done before compilation can continue.
For example, zstd-sys takes really a long time to compile zstd and all crates that depend on it have to wait until they are compiled.
Given that bindgen usually doesn't take that long (if user enabled zstd-sys/bindgen), the dependents of zstd-sys could start compilation earlier.
Proposed Solution
I propose that we introduce a new flag
cargo::no-new-rust-code-change, it would tell cargo that the build script won't do anything that affects compilation of the rust code of this crate (generating code, issuing cfgs, etc), and so the rust crate can generate the rmeta and compile the code.Once the rmeta is generated, dependents of this crate can start compiling without waiting for the whole compilation to be done.
If the build-script violates the protocol and actually tries to issue build-script instructions for cfgs then compilation would fail.
Generating code or changing already generated code that the rust crate
includes!()would be considered the same as trying to modify the source code of the crate when rustc is compiling, rustc/cargo does not guarantee whether it would be read, and rustc could read the files when build script is updating it.implementation
Implementing this would need
-Zno-linkto defer archiving to-Zlink-onlyso that it doesn't rely on external static libraries produced by the build script.Notes
how would that benefit systems with less than 4 cores, e.g. GHA?
It definitely cannot benefit single-core system, however for system with 2-4 cores it could still benefit.
build-script using cc-rs
P.S. I am one of the maintainers of cc-rs
For crate using cc-rs to compile the code, cc-rs cannot fully utilise every core at all time.
Particularly, cc-rs runs
is_flag_supportedlogic in sequential, and that logic actually tries to compile a dummy c file to check if the compiler supports the flag.cc-rs also has other places where it has to run external command sequentially to retrieve information (checking compiler family, looking for macOS/windows env, etc).
During parallel compilation (if feature
cc/parallelis enabled), cc is not guaranteed to spawn a new job immediately when a job token is freed by another process (e.g. other build scripts, rustc, etc).compilation using external systems
Compilation using external systems such as configure + make, or cmake (potentially through cmake-rs) typically involves a setup/env detection and makefile/ninja generation process that AFAIK is single thread and takes quite some time.
It might be enough for rustc to compile the crate itself and unblock the dependents to be compiled in parallel.
And since ninja/make has an internal dependencies resolution process, it is not likely to fully utilise all cores at all time.