Rust compiler fails to build when host linker requires extra arguments

[ComputerDruid]

Using a config.toml, I've configured a custom host linker (building on a Mac):

[target.x86_64-apple-darwin]
cc = "/opt/s/w/ir/x/w/cipd/bin/clang"
cxx = "/opt/s/w/ir/x/w/cipd/bin/clang++"
ar = "/opt/s/w/ir/x/w/cipd/bin/llvm-ar"
ranlib = "/opt/s/w/ir/x/w/cipd/bin/llvm-ranlib"
profiler = true
linker = "/opt/s/w/ir/x/w/cipd/bin/clang++"

full config.toml

On this machine, the linker requires a custom --sysroot to be passed to it, so I also set some environment variables:

CFLAGS_x86_64_apple_darwin=--target=x86_64-apple-darwin --sysroot=/opt/s/w/ir/cache/macos_sdk/XCode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk
CXXFLAGS_x86_64_apple_darwin=--target=x86_64-apple-darwin --sysroot=/opt/s/w/ir/cache/macos_sdk/XCode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk
LDFLAGS_x86_64_apple_darwin=--target=x86_64-apple-darwin --sysroot=/opt/s/w/ir/cache/macos_sdk/XCode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk
CARGO_TARGET_X86_64_APPLE_DARWIN_RUSTFLAGS=-C link-arg=--target=x86_64-apple-darwin -C link-arg=--sysroot=/opt/s/w/ir/cache/macos_sdk/XCode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk

full environment

But this isn't enough to ensure that --sysroot is always passed to the linker, because when I try to build the compiler with ./x.py install, I get:

 error: linking with `/opt/s/w/ir/x/w/cipd/bin/clang++` failed: exit status: 1
   |
   = note: "/opt/s/w/ir/x/w/cipd/bin/clang++" "-m64" "-arch" "x86_64" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/bootstrap/debug/build/serde_json-a9e1ff0fe4c1cd01/build_script_build-a9e1ff0fe4c1cd01.build_script_build.8b6a5296-cgu.0.rcgu.o" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/bootstrap/debug/build/serde_json-a9e1ff0fe4c1cd01/build_script_build-a9e1ff0fe4c1cd01.build_script_build.8b6a5296-cgu.1.rcgu.o" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/bootstrap/debug/build/serde_json-a9e1ff0fe4c1cd01/build_script_build-a9e1ff0fe4c1cd01.build_script_build.8b6a5296-cgu.10.rcgu.o" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/bootstrap/debug/build/serde_json-a9e1ff0fe4c1cd01/build_script_build-a9e1ff0fe4c1cd01.build_script_build.8b6a5296-cgu.11.rcgu.o" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/bootstrap/debug/build/serde_json-a9e1ff0fe4c1cd01/build_script_build-a9e1ff0fe4c1cd01.build_script_build.8b6a5296-cgu.12.rcgu.o" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/bootstrap/debug/build/serde_json-a9e1ff0fe4c1cd01/build_script_build-a9e1ff0fe4c1cd01.build_script_build.8b6a5296-cgu.13.rcgu.o" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/bootstrap/debug/build/serde_json-a9e1ff0fe4c1cd01/build_script_build-a9e1ff0fe4c1cd01.build_script_build.8b6a5296-cgu.14.rcgu.o" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/bootstrap/debug/build/serde_json-a9e1ff0fe4c1cd01/build_script_build-a9e1ff0fe4c1cd01.build_script_build.8b6a5296-cgu.15.rcgu.o" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/bootstrap/debug/build/serde_json-a9e1ff0fe4c1cd01/build_script_build-a9e1ff0fe4c1cd01.build_script_build.8b6a5296-cgu.2.rcgu.o" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/bootstrap/debug/build/serde_json-a9e1ff0fe4c1cd01/build_script_build-a9e1ff0fe4c1cd01.build_script_build.8b6a5296-cgu.3.rcgu.o" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/bootstrap/debug/build/serde_json-a9e1ff0fe4c1cd01/build_script_build-a9e1ff0fe4c1cd01.build_script_build.8b6a5296-cgu.4.rcgu.o" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/bootstrap/debug/build/serde_json-a9e1ff0fe4c1cd01/build_script_build-a9e1ff0fe4c1cd01.build_script_build.8b6a5296-cgu.5.rcgu.o" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/bootstrap/debug/build/serde_json-a9e1ff0fe4c1cd01/build_script_build-a9e1ff0fe4c1cd01.build_script_build.8b6a5296-cgu.6.rcgu.o" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/bootstrap/debug/build/serde_json-a9e1ff0fe4c1cd01/build_script_build-a9e1ff0fe4c1cd01.build_script_build.8b6a5296-cgu.7.rcgu.o" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/bootstrap/debug/build/serde_json-a9e1ff0fe4c1cd01/build_script_build-a9e1ff0fe4c1cd01.build_script_build.8b6a5296-cgu.8.rcgu.o" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/bootstrap/debug/build/serde_json-a9e1ff0fe4c1cd01/build_script_build-a9e1ff0fe4c1cd01.build_script_build.8b6a5296-cgu.9.rcgu.o" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/bootstrap/debug/build/serde_json-a9e1ff0fe4c1cd01/build_script_build-a9e1ff0fe4c1cd01.2u71r0gxx5cfeky5.rcgu.o" "-L" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/bootstrap/debug/deps" "-L" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-apple-darwin/stage0/lib/rustlib/x86_64-apple-darwin/lib" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-apple-darwin/stage0/lib/rustlib/x86_64-apple-darwin/lib/libstd-301ac569e1f5b4aa.rlib" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-apple-darwin/stage0/lib/rustlib/x86_64-apple-darwin/lib/libpanic_unwind-16e99e032d9da8c8.rlib" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-apple-darwin/stage0/lib/rustlib/x86_64-apple-darwin/lib/libobject-ee624fb1c42ac415.rlib" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-apple-darwin/stage0/lib/rustlib/x86_64-apple-darwin/lib/libmemchr-3d9a363ff0815197.rlib" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-apple-darwin/stage0/lib/rustlib/x86_64-apple-darwin/lib/libaddr2line-90c612fcf2db3a6c.rlib" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-apple-darwin/stage0/lib/rustlib/x86_64-apple-darwin/lib/libgimli-0662566af5cce885.rlib" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-apple-darwin/stage0/lib/rustlib/x86_64-apple-darwin/lib/libstd_detect-6b4fbea665e8ba50.rlib" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-apple-darwin/stage0/lib/rustlib/x86_64-apple-darwin/lib/librustc_demangle-1acdb4532b0dd3e5.rlib" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-apple-darwin/stage0/lib/rustlib/x86_64-apple-darwin/lib/libhashbrown-95671edf95d63cce.rlib" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-apple-darwin/stage0/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_alloc-0470582744ac5371.rlib" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-apple-darwin/stage0/lib/rustlib/x86_64-apple-darwin/lib/libunwind-cce6b5d0dd51e55f.rlib" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-apple-darwin/stage0/lib/rustlib/x86_64-apple-darwin/lib/libcfg_if-b05763aba30b2386.rlib" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-apple-darwin/stage0/lib/rustlib/x86_64-apple-darwin/lib/liblibc-598373f5a98f0c63.rlib" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-apple-darwin/stage0/lib/rustlib/x86_64-apple-darwin/lib/liballoc-21a9c5a08f6fb9ac.rlib" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-apple-darwin/stage0/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_core-7cdb23da1434d3f8.rlib" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-apple-darwin/stage0/lib/rustlib/x86_64-apple-darwin/lib/libcore-6e3f464b02c07d39.rlib" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-apple-darwin/stage0/lib/rustlib/x86_64-apple-darwin/lib/libcompiler_builtins-3b25963d2d3ed4ab.rlib" "-lSystem" "-lresolv" "-lc" "-lm" "-liconv" "-L" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-apple-darwin/stage0/lib/rustlib/x86_64-apple-darwin/lib" "-o" "/opt/s/w/ir/x/w/staging/build/fuchsia-build/bootstrap/debug/build/serde_json-a9e1ff0fe4c1cd01/build_script_build-a9e1ff0fe4c1cd01" "-Wl,-dead_strip" "-nodefaultlibs"
   = note: ld: library not found for -lSystem
           clang-14: error: linker command failed with exit code 1 (use -v to see invocation)

full build output

And the resulting linker command does not include --sysroot.

To make it easier to reproduce, I created the following script:

#!/bin/bash

found=0
for arg in "$@"; do
  if [[ "$arg" =~ --sysroot ]]; then
    found=1
    break
  fi
done
if [ $found -eq 0 ]; then
  echo "linker argv missing sysroot" >&2
  exit 1
fi

# or whatever linker you would normally use
clang++ "$@"

Which can be used as the linker in config.toml to reproduce this, because it artificially requires the extra command-line argument. On a mac I believe you can get the correct value for the sysroot with xcrun --sdk macosx --show-sdk-path, on linux I believe / is a reasonable sysroot to pass.

I actually tried patching boostrap.py:

From 05c21525247f4d35b82b8828efff43f26caaf4aa Mon Sep 17 00:00:00 2001
From: Dan Johnson <computerdruid@google.com>
Date: Thu, 10 Feb 2022 15:41:12 -0800
Subject: [PATCH] WIP use target-specific rustflags when building bootstrap

Change-Id: If6ad55f67e77c5698854f600c873fadbbfbbd8ae
---
 src/bootstrap/bootstrap.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/src/bootstrap/bootstrap.py b/src/bootstrap/bootstrap.py
index 86115a90294..9a390b783af 100644
--- a/src/bootstrap/bootstrap.py
+++ b/src/bootstrap/bootstrap.py
@@ -999,6 +999,10 @@ class RustBuild(object):
 
         # preserve existing RUSTFLAGS
         env.setdefault("RUSTFLAGS", "")
+        envify = lambda b: b.replace("-", "_").upper()
+        target_rustflags = env.get("CARGO_TARGET_%s_RUSTFLAGS"%envify(self.build))
+        if target_rustflags:
+            env["RUSTFLAGS"] += " " + target_rustflags
         build_section = "target.{}".format(self.build)
         target_features = []
         if self.get_toml("crt-static", build_section) == "true":
-- 
2.35.1.265.g69c8d7142f-goog

Which gets it a little bit further, but then it fails while building build scripts during the libstd build.

Meta

I noticed this while trying to build the latest commit at the time, which happened to be 9a60099

[tmandry]

I talked with @ComputerDruid about this some. After the initial bootstrap.py patch, it looks like we hit rust-lang/cargo#4423.

[tmandry]

Reading into the cargo bug some more, I found some options that would help.

• Tracking Issue for target-applies-to-host cargo#9453
• Tracking Issue for host-config cargo#9452

Either would allow us to configure the flags passed to build scripts, one using the "normal" RUSTFLAGS for that target as I understand it, and the other using a dedicated section in .cargo/config.

The simplest change that would fix our use case is probably to expose target-applies-to-host as an option in the build config and forward that to cargo. Then the RUSTFLAGS we are supplying for the host triple would be applied.

Ideally there would be a way to set flags for targets directly in the rustbuild config.toml, and we could extend that to a host section as well.

[jyn514]

I think -Ztarget-applies-to-host works today (you can pass it with CARGOFLAGS). You could also try setting MAGIC_EXTRA_RUSTFLAGS, I believe those apply to build scripts:

rust/src/bootstrap/bin/rustc.rs

Line 156 in 471fa05

if let Ok(flags) = env::var("MAGIC_EXTRA_RUSTFLAGS") {

I'm going to close this since it's not an issue with bootstrap itself.