Skip to content

Compile time+memory regression between 1.49.0 and 1.50.0 #84873

Open
Open
Compile time+memory regression between 1.49.0 and 1.50.0#84873
Labels
C-bugCategory: This is a bug.I-compilememIssue: Problems and improvements with respect to memory usage during compilation.I-compiletimeIssue: Problems and improvements with respect to compile times.P-highHigh priorityT-compilerRelevant to the compiler team, which will review and decide on the PR/issue.regression-from-stable-to-stablePerformance or correctness regression from one stable version to another.
@olix0r

Description

@olix0r
olix0r
opened on May 3, 2021

Our compile times and memory footprint regressed substantially between Rust 1.49.0 and 1.50.0.

This regression is a hard blocker for us to be able to upgrade our project from 1.49.0. As this is a severe issue for us, please let us know if there's anything we can do to provide more diagnostics, test changes, etc.

This may be related to other recent issues that describe similar behavior:

Code

This regression is observed on recent versions of the Linkerd proxy. It's known that the proxy can manifest large type signatures, so builds disable debug symbols by default

This regression is not obvious with other, smaller projects that we maintain, so I'm unable to provide a smaller repro.

Version it worked on

With Rust 1.49.0, the binary compiles in a little over two minutes, using a little over 1GB of memory:

cargo clean && /usr/bin/time -v cargo +1.49.0 build -p linkerd2-proxy

...
    Finished dev [unoptimized] target(s) in 2m 12s
        Command being timed: "cargo +1.49.0 build -p linkerd2-proxy"
        User time (seconds): 326.00
        System time (seconds): 14.55
        Percent of CPU this job got: 256%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 2:12.78
...
        Maximum resident set size (kbytes): 1308088
...
        Exit status: 0

Version with regression

Using Rust 1.50.0, rustc runs for about 40 minutes before exhausting the system's memory:

cargo clean && /usr/bin/time -v cargo +1.50.0 build -p linkerd2-proxy

  process didn't exit successfully: `rustc --crate-name linkerd2_proxy --edition=2018 linkerd2-proxy/src/main.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no --cfg 'feature="default"' --cfg 'feature="multicore"' --cfg 'feature="num_cpus"' -C metadata=c2caa01c24c285f7 -C extra-filename=-c2caa01c24c285f7 --out-dir /home/ver/b/linkerd2-proxy/target/debug/deps -C incremental=/home/ver/b/linkerd2-proxy/target/debug/incremental -L dependency=/home/ver/b/linkerd2-proxy/target/debug/deps --extern futures=/home/ver/b/linkerd2-proxy/target/debug/deps/libfutures-01f6240a9e4201d3.rlib --extern linkerd_app=/home/ver/b/linkerd2-proxy/target/debug/deps/liblinkerd_app-dd18fcabeef913ae.rlib --extern linkerd_signal=/home/ver/b/linkerd2-proxy/target/debug/deps/liblinkerd_signal-14263c92fcb0fc0b.rlib --extern num_cpus=/home/ver/b/linkerd2-proxy/target/debug/deps/libnum_cpus-74ee144d51bc047e.rlib --extern tokio=/home/ver/b/linkerd2-proxy/target/debug/deps/libtokio-de277ade05efd960.rlib --extern tracing=/home/ver/b/linkerd2-proxy/target/debug/deps/libtracing-03fd6826b817dabe.rlib -L native=/home/ver/b/linkerd2-proxy/target/debug/build/ring-a8b802a4aa425398/out` (signal: 9, SIGKILL: kill)
Command exited with non-zero status 101
	Command being timed: "cargo +1.50.0 build -p linkerd2-proxy"
	User time (seconds): 2986.18
	System time (seconds): 101.73
	Percent of CPU this job got: 133%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 38:29.28
...
	Maximum resident set size (kbytes): 62953524
...
	Exit status: 101

rustc --version --verbose:

rustc 1.50.0 (cb75ad5db 2021-02-10)
binary: rustc
commit-hash: cb75ad5db02783e8b0222fee363c5f63f7e2cf5b
commit-date: 2021-02-10
host: x86_64-unknown-linux-gnu
release: 1.50.0

We see similar behavior with more recent versions of Rust as well, including 1.51.0 and nightly (cargo 1.53.0-nightly (0ed318d18 2021-04-23)) as well.

cc @hawkw, who can provide some more details. I believe we've tested with lto=off without any changes in behavior.

@rustbot modify labels: +regression-from-stable-to-stable -regression-untriaged

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-bugCategory: This is a bug.I-compilememIssue: Problems and improvements with respect to memory usage during compilation.I-compiletimeIssue: Problems and improvements with respect to compile times.P-highHigh priorityT-compilerRelevant to the compiler team, which will review and decide on the PR/issue.regression-from-stable-to-stablePerformance or correctness regression from one stable version to another.

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions