Skip to content

Introduce -Zembed-metadata to allow omitting full metadata from rlibs and dylibs #137535

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Apr 1, 2025

Conversation

Kobzol
Copy link
Contributor

@Kobzol Kobzol commented Feb 24, 2025

This is a continuation of #120855 (I was mentored by @bjorn3 to move it forward). Most of the original code was written by bjorn3, I tried to clean it up a bit and add some documentation and tests.

This PR introduces a new unstable compiler flag called -Zembed-metadata=[no|yes], with the default being yes (see #57076 for context). When set to no, rustc will only store a small metadata stub inside rlibs/dylibs instead of the full metadata, to keep their size smaller. It should be used in combination with --emit=metadata, so that the users of such a compiled library can still read the metadata from the corresponding .rmeta file. This comment shows an example of binary/artifact size wins that can be achieved using this approach.

Contrary to #120855, this PR only introduces the new flag, along with a couple of run-make tests and documentation, but does not yet use it in bootstrap to actually compile rustc. I plan to do that as a follow-up step (along with integration in Cargo, which should ideally just always pass this flag to reduce the size of target directories).

Fixes #23366
Closes #29511
Fixes #57076

Another attempt of #93945 and #120855.

r? @petrochenkov

@rustbot rustbot added A-run-make Area: port run-make Makefiles to rmake.rs S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Feb 24, 2025
@rust-log-analyzer

This comment has been minimized.

@Kobzol
Copy link
Contributor Author

Kobzol commented Feb 24, 2025

(The test failure is the one situation not yet implemented, which is described in the PR description - --extern plus .rmeta in a different directory than the rlib/dylib).

@petrochenkov
Copy link
Contributor

I found it quite surprising that when --extern is passed, the -L search logic is completely skipped

That's by design, --extern with paths disables all the directory search, makes the build system own the input files, and makes rustc just do what the build system tells it.
I'd actually consider this the primary mode of work for rustc, and all the directory search is a sort of legacy (well, except -L dependency=PATH).

--extern supports passing multiple paths for the same crate, to specify all its rmeta, rlib and dylib versions.
So in case of split metadata I'd also expect cargo (or another build system) to pass path to the split rmeta explicitly, and rustc to not do any heuristics.

In #120855 the heuristics were a temporary measure to enable split metadata in bootstrap, while cargo still doesn't support it.
It should be fine to add them in your follow up PR implementing the bootstrap changes as well, but in the long run I don't think they should exist.

@petrochenkov petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 13, 2025
@Kobzol
Copy link
Contributor Author

Kobzol commented Mar 13, 2025

Interesting. So if I understood it correctly, you would like to remove all heuristics from rustc (so no lookup of .rmeta in known library search dirs and no ".rmeta in the same directory as .rlib" heuristic) and let Cargo pass essentially both .rlib and .rmeta for every dependency through --extern? I guess that's cleaner in a sense.

On the other hand, it would essentially double the length of the rustc command line, because for every library that you depend on, Cargo would have to pass both .rmeta and .rlib. Do you think that wouldn't be an issue? I suppose that it would only happen for the leaf linked artifact, because the intermediate libraries seem to only receive .rmeta files from Cargo, while the final artifact receives .rlib.

@petrochenkov
Copy link
Contributor

So if I understood it correctly, you would like to remove all heuristics from rustc...

Yes.

On the other hand, it would essentially double the length of the rustc command line, because for every library that you depend on, Cargo would have to pass both .rmeta and .rlib.

My impression was that it is already the case with pipelined builds, but apparently it's indeed only .rmetas for dependencies and only .rlibs for the root crate.
I think it's fine if it happens only for direct dependencies of binary crates, in any case we can reconsider heuristics if there are any complaints about this.

@rustbot
Copy link
Collaborator

rustbot commented Mar 14, 2025

Some changes occurred in compiler/rustc_codegen_ssa

cc @WaffleLapkin

This PR modifies run-make tests.

cc @jieyouxu

@Kobzol
Copy link
Contributor Author

Kobzol commented Mar 14, 2025

Ok, rebased and removed the heuristics. I added an error message when the full metadata is not found, but I'm not sure if we should emit the error eagerly (left a comment in code). Could you please take a look at fe29fec? Thank you!

@rust-log-analyzer

This comment has been minimized.

@Kobzol
Copy link
Contributor Author

Kobzol commented Mar 28, 2025

rust-lang/compiler-team#851 was accepted, so I think that we can go forward with this.

@petrochenkov What do you think about renaming the flag to -Zembed-metadata=[no|yes], with yes being the default, to be consistent with -Cembed-bitcode=[no|yes], which is a bit similar in spirit?

@petrochenkov
Copy link
Contributor

What do you think about renaming the flag to -Zembed-metadata=[no|yes], with yes being the default, to be consistent with -Cembed-bitcode=[no|yes], which is a bit similar in spirit?

Seems ok to me.

@Kobzol Kobzol changed the title Introduce -Zsplit-metadata to split metadata out of rlibs/dylibs Introduce -Zembed-metadata to allow omitting full metadata from rlibs and dylibs Mar 28, 2025
@Kobzol
Copy link
Contributor Author

Kobzol commented Mar 28, 2025

Ok, changed it to -Zembed-metadata=[no|yes].

@rustbot ready

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Mar 28, 2025
@rust-log-analyzer

This comment has been minimized.

@Kobzol
Copy link
Contributor Author

Kobzol commented Mar 31, 2025

Created a tracking issue and fixed tests.

@petrochenkov
Copy link
Contributor

r=me with the metadata version updated.
@rustbot author

@rustbot rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 31, 2025
@rust-log-analyzer

This comment has been minimized.

@rustbot
Copy link
Collaborator

rustbot commented Apr 1, 2025

rust-analyzer is developed in its own repository. If possible, consider making this change to rust-lang/rust-analyzer instead.

cc @rust-lang/rust-analyzer

@Kobzol
Copy link
Contributor Author

Kobzol commented Apr 1, 2025

@bors r=petrochenkov rollup=never

Marked as rollup=never, because this modifies metadata version.

@bors
Copy link
Collaborator

bors commented Apr 1, 2025

📌 Commit f0efb97 has been approved by petrochenkov

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Apr 1, 2025
@bors
Copy link
Collaborator

bors commented Apr 1, 2025

⌛ Testing commit f0efb97 with merge 8c35f4a...

@bors
Copy link
Collaborator

bors commented Apr 1, 2025

☀️ Test successful - checks-actions
Approved by: petrochenkov
Pushing 8c35f4a to master...

@bors bors added the merged-by-bors This PR was explicitly merged by bors. label Apr 1, 2025
@bors bors merged commit 8c35f4a into rust-lang:master Apr 1, 2025
7 checks passed
@rustbot rustbot added this to the 1.88.0 milestone Apr 1, 2025
Copy link

github-actions bot commented Apr 1, 2025

What is this? This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing ed20157 (parent) -> 8c35f4a (this PR)

Test differences

Show 43 test diffs

Stage 1

  • errors::verify_metadata_crate_location_unknown_type_68: pass -> [missing] (J0)
  • errors::verify_metadata_crate_location_unknown_type_69: [missing] -> pass (J0)
  • errors::verify_metadata_dl_error_63: pass -> [missing] (J0)
  • errors::verify_metadata_dl_error_64: [missing] -> pass (J0)
  • errors::verify_metadata_found_staticlib_66: pass -> [missing] (J0)
  • errors::verify_metadata_found_staticlib_67: [missing] -> pass (J0)
  • errors::verify_metadata_full_metadata_not_found_61: [missing] -> pass (J0)
  • errors::verify_metadata_import_name_type_form_71: pass -> [missing] (J0)
  • errors::verify_metadata_import_name_type_form_72: [missing] -> pass (J0)
  • errors::verify_metadata_import_name_type_raw_74: pass -> [missing] (J0)
  • errors::verify_metadata_import_name_type_raw_75: [missing] -> pass (J0)
  • errors::verify_metadata_import_name_type_x86_72: pass -> [missing] (J0)
  • errors::verify_metadata_import_name_type_x86_73: [missing] -> pass (J0)
  • errors::verify_metadata_incompatible_rustc_67: pass -> [missing] (J0)
  • errors::verify_metadata_incompatible_rustc_68: [missing] -> pass (J0)
  • errors::verify_metadata_incompatible_target_modifiers_76: pass -> [missing] (J0)
  • errors::verify_metadata_incompatible_target_modifiers_77: [missing] -> pass (J0)
  • errors::verify_metadata_incompatible_target_modifiers_l_missed_77: pass -> [missing] (J0)
  • errors::verify_metadata_incompatible_target_modifiers_l_missed_78: [missing] -> pass (J0)
  • errors::verify_metadata_incompatible_target_modifiers_r_missed_78: pass -> [missing] (J0)
  • errors::verify_metadata_incompatible_target_modifiers_r_missed_79: [missing] -> pass (J0)
  • errors::verify_metadata_lib_filename_form_69: pass -> [missing] (J0)
  • errors::verify_metadata_lib_filename_form_70: [missing] -> pass (J0)
  • errors::verify_metadata_multiple_import_name_type_70: pass -> [missing] (J0)
  • errors::verify_metadata_multiple_import_name_type_71: [missing] -> pass (J0)
  • errors::verify_metadata_newer_crate_version_64: pass -> [missing] (J0)
  • errors::verify_metadata_newer_crate_version_65: [missing] -> pass (J0)
  • errors::verify_metadata_no_crate_with_triple_65: pass -> [missing] (J0)
  • errors::verify_metadata_no_crate_with_triple_66: [missing] -> pass (J0)
  • errors::verify_metadata_stable_crate_id_collision_62: pass -> [missing] (J0)
  • errors::verify_metadata_stable_crate_id_collision_63: [missing] -> pass (J0)
  • errors::verify_metadata_symbol_conflicts_current_61: pass -> [missing] (J0)
  • errors::verify_metadata_symbol_conflicts_current_62: [missing] -> pass (J0)
  • errors::verify_metadata_unknown_import_name_type_73: pass -> [missing] (J0)
  • errors::verify_metadata_unknown_import_name_type_74: [missing] -> pass (J0)
  • errors::verify_metadata_unknown_target_modifier_unsafe_allowed_79: pass -> [missing] (J0)
  • errors::verify_metadata_unknown_target_modifier_unsafe_allowed_80: [missing] -> pass (J0)
  • errors::verify_metadata_wasm_c_abi_75: pass -> [missing] (J0)
  • errors::verify_metadata_wasm_c_abi_76: [missing] -> pass (J0)
  • [run-make] tests/run-make/embed-metadata: [missing] -> pass (J1)

Stage 2

  • [run-make] tests/run-make/embed-metadata: [missing] -> pass (J2)

Additionally, 2 doctest diffs were found. These are ignored, as they are noisy.

Job group index

  • J0: aarch64-apple, aarch64-gnu, i686-gnu-2, i686-gnu-nopt-2, i686-mingw-2, i686-mingw-3, i686-msvc-2, x86_64-apple-1, x86_64-gnu, x86_64-gnu-llvm-18-3, x86_64-gnu-llvm-19-3, x86_64-gnu-nopt, x86_64-gnu-stable, x86_64-mingw-2, x86_64-msvc-2
  • J1: x86_64-gnu-llvm-18-3, x86_64-gnu-llvm-19-3
  • J2: aarch64-apple, aarch64-gnu, aarch64-gnu-debug, arm-android, armhf-gnu, dist-i586-gnu-i586-i686-musl, dist-various-1, i686-gnu-1, i686-gnu-nopt-1, i686-msvc-1, test-various, x86_64-apple-1, x86_64-gnu, x86_64-gnu-debug, x86_64-gnu-llvm-18-1, x86_64-gnu-llvm-18-2, x86_64-gnu-llvm-19-1, x86_64-gnu-llvm-19-2, x86_64-gnu-nopt, x86_64-gnu-stable, x86_64-mingw-1, x86_64-msvc-1

Job duration changes

  1. aarch64-apple: 3668.0s -> 4135.9s (12.8%)
  2. dist-x86_64-apple: 8228.8s -> 9021.5s (9.6%)
  3. x86_64-gnu-aux: 6050.3s -> 6564.3s (8.5%)
  4. x86_64-apple-2: 5146.4s -> 5562.5s (8.1%)
  5. i686-gnu-2: 6154.4s -> 6543.8s (6.3%)
  6. x86_64-mingw-1: 9132.0s -> 9660.3s (5.8%)
  7. i686-msvc-2: 7121.3s -> 7515.9s (5.5%)
  8. dist-x86_64-musl: 7165.8s -> 7498.9s (4.6%)
  9. dist-x86_64-msvc: 6103.3s -> 6346.6s (4.0%)
  10. mingw-check: 1251.6s -> 1295.7s (3.5%)
How to interpret the job duration changes?

Job durations can vary a lot, based on the actual runner instance
that executed the job, system noise, invalidated caches, etc. The table above is provided
mostly for t-infra members, for simpler debugging of potential CI slow-downs.

@rust-timer
Copy link
Collaborator

Finished benchmarking commit (8c35f4a): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results (secondary -1.7%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)
- - 0
Regressions ❌
(secondary)
- - 0
Improvements ✅
(primary)
- - 0
Improvements ✅
(secondary)
-1.7% [-1.8%, -1.6%] 2
All ❌✅ (primary) - - 0

Cycles

Results (secondary 3.9%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)
- - 0
Regressions ❌
(secondary)
3.9% [3.9%, 3.9%] 1
Improvements ✅
(primary)
- - 0
Improvements ✅
(secondary)
- - 0
All ❌✅ (primary) - - 0

Binary size

Results (secondary 0.0%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)
- - 0
Regressions ❌
(secondary)
0.0% [0.0%, 0.0%] 1
Improvements ✅
(primary)
- - 0
Improvements ✅
(secondary)
- - 0
All ❌✅ (primary) - - 0

Bootstrap: 773.019s -> 774.818s (0.23%)
Artifact size: 365.92 MiB -> 365.93 MiB (0.00%)

github-merge-queue bot pushed a commit to rust-lang/cargo that referenced this pull request May 6, 2025
### What does this PR try to resolve?

This PR adds Cargo integration for the new unstable `-Zembed-metadata`
rustc flag, which was implemented in
rust-lang/rust#137535 ([tracking
issue](rust-lang/rust#139165)). The new
behavior has to be enabled explicitly using a new unstable CLI flag
`-Zno-embed-metadata`.

The `-Zembed-metadata=no` rustc flag can reduce disk usage of compiled
artifacts, and also the size of Rust dynamic library artifacts shipped
to users. However, it is not enough to just pass this flag through
`RUSTFLAGS`; it needs to be integrated within Cargo, because it
interacts with how the `--emit` flag is passed to rustc, and also how
`--extern` args are passed to the final linked artifact build by Cargo.
Furthermore, using the flag for all crates in a crate graph compiled by
Cargo would be suboptimal (this will all be described below).

When you pass `-Zembed-metadata=no` to rustc, it will not store Rust
metadata into the compiled artifact. This is important when compiling
libs/rlibs/dylibs, since it reduces their size on disk. However, this
also means that everytime we use this flag, we have to make sure that we
also:
- Include `metadata` in the `--emit` flag to generate a `.rmeta` file,
otherwise no metadata would be generated whatsoever, which would mean
that the artifact wouldn't be usable as a dependency.
- Pass also `--extern <dep>=<path>.rmeta` when compiling the final
linkable artifact. Before, Cargo would only pass `--extern
<dep>=<path>.[rlib|so|dll]`. Since with `-Zembed-metadata=no`, the
metadata is only in the `.rmeta` file and not in the rlib/dylib, this is
needed to help rustc find out where the metadata lies.
- Note: this essentially doubles the cmdline length when compiling the
final linked artifact. Not sure if that is a concern.

The two points above is what this PR implements, and why this rustc flag
needs Cargo integration.

The `-Zembed-metadata` flag is only passed to libs, rlibs and dylibs. It
does not seem to make sense for other crate types. The one situation
where it might make sense are proc macros, but according to @bjorn3 (who
initially came up with the idea for `-Zembed-metadata`, it isn't really
worth it).

Here is a table that summarizes the changes in passed flags and
generated files on disk for rlibs and dylibs:

| **Crate type** | **Flags** | **Generated files** | **Disk usage** |
|--|--|--|--|
| Rlib/Lib (before) | `--emit=dep-info,metadata,link` | `.rlib` (with
metadata), `.rmeta` (for pipelining) | - |
| Rlib/Lib (after) | `--emit=dep-info,metadata,link -Zembed-metadata=no`
| `.rlib` (without metadata), `.rmeta` (for metadata/pipelining) |
Reduced (metadata no longer duplicated) |
| Dylib (before) | `--emit=dep-info,link` | `[.so\|.dll]` (with
metadata) | - |
| Dylib (after) | `--emit=dep-info,metadata,link -Zembed-metadata=no` |
`[.so\|.dll]` (without metadata), `.rmeta` | Unchanged, but split
between two files |

Behavior for other target kinds/crate types should be unchanged.

From the table above, we can see two benefits of using
`-Zembed-metadata=no`:
- For rlibs/dylibs, we no longer store their metadata twice in the
target directory, thus reducing target directory size.
- For dylibs, we store esssentially the same amount of data on disk, but
the benefit is that the metadata is now in a separate .rmeta file. This
means that you can ship the dylib (`.so`/`.dll`) to users without also
shipping the metadata. This would slightly reduce e.g. the
[size](rust-lang/rust#120855 (comment))
of the shipped rustc toolchains (note that the size reduction here is
after the toolchain has been already heavily compressed).

Note that if this behavior ever becomes the default, it should be
possible to simplify the code quite a bit, and essentially merge the
`requires_upstream_objects` and `benefits_from_split_metadata`
functions.

I did a very simple initial benchmark to evaluate the space savings on
cargo itself and
[hyperqueue](https://github.com/It4innovations/hyperqueue) (a mid-size
crate from my work) using `cargo build` and `cargo build --release` with
and without `-Zembed-metadata=no`:

![image](https://github.com/user-attachments/assets/a26994a2-156f-4863-a823-1042ebe03bf0)

For debug/incremental builds, the effect is smaller, as the artifact
disk usage is dwarfed by incremental artifacts and debuginfo. But for
(non-incremental) release builds, the disk savings (and also performed
I/O operations) are significantly reduced.

### How should we test and review this PR?

I wrote two basic tests. The second one tests a situation where a crate
depends on a dylib dependency, which is quite rare, but the behavior of
this has actually changed in this PR (see comparison table above).
Testing this on various real-world projects (or even trying to enable it
by default across the whole Cargo suite?) might be beneficial.

## Unresolved questions

### Is this a breaking change?
With this new behavior, dylibs and rlibs will no longer contain
metadata. If they are compiled with Cargo, that shouldn't matter, but
other build systems might have to adapt.

### Should this become the default?
I think that in terms of disk size usage and performed I/O operations,
it is a pure win. It should either generate less disk data (for rlibs)
or the ~same amount of data for dylibs (the data will be a bit larger,
because the dylib will still contain a metadata stub header, but that's
like 50 bytes and doesn't scale with the size of the dylib, so it's
negligible).

So I think that eventually, we should just do this by default in Cargo,
unless some concerns are found. I suppose that before stabilizing we
should also benchmark the effect on compilation performance.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-run-make Area: port run-make Makefiles to rmake.rs merged-by-bors This PR was explicitly merged by bors. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
8 participants