Skip to content

Build a native arm64 image instead of emulating amd64 - #201

Open
aleksanderllada wants to merge 1 commit into
mainfrom
aarruda/native-arm64-docker-build
Open

Build a native arm64 image instead of emulating amd64#201
aleksanderllada wants to merge 1 commit into
mainfrom
aarruda/native-arm64-docker-build

Conversation

@aleksanderllada

Copy link
Copy Markdown
Contributor

Summary

An arm64 image built from source SIGILLs on anything short of Graviton3+, because .cargo/config.toml pins -C target-cpu=neoverse-512tvb for aarch64-unknown-linux-gnu. DOCKER.md worked around it by telling Apple Silicon users to build --platform linux/amd64 and emulate:

- On Apple Silicon (M-series Macs), builds must target `linux/amd64` due to Graviton-specific
  compiler flags in `.cargo/config.toml` for `aarch64-unknown-linux-gnu`

This builds natively instead. docker build -f lore-server/Dockerfile . now produces a baseline armv8-a image on Apple Silicon, and --build-arg ARM64_TARGET_CPU=neoverse-512tvb opts back into the tuning for a Graviton deployment.

Two things worth a reviewer's attention

Overriding the pin needs RUSTFLAGS specifically, not CARGO_TARGET_<triple>_RUSTFLAGS. The latter is only another source for the same config key, and cargo joins config arrays — the config's -C target-cpu would survive and the build would still fault. RUSTFLAGS replaces [build] and [target.*] outright, which is why the flags are restated in full rather than appended to.

RUSTFLAGS does not reach a C compiler. lore-base/build.rs pins the same -mcpu for its cc build of rpmalloc, so the Rust half alone is not enough. That flag raises the architecture floor as well as the tuning — disassembling both builds with GCC 14:

only in the tuned build:  casa casal casl ldadd ldapr madd stlur

No SVE, so Apple Silicon was never at risk from the C half. But stlur is FEAT_LRCPC2 (armv8.4) and ldapr is armv8.3, and Neoverse N1 has neither — so an otherwise-baseline image would still have faulted in the allocator on Ampere Altra and Graviton2. build.rs now reads LORE_ARM64_TARGET_CPU, defaulting to the value it hardcoded so nothing else changes, and the Dockerfile passes the choice through.

Also here: EXPOSE defaults to TCP, so it was not declaring the QUIC listener sharing port 41337 — docker run -P left it unreachable.

Test evidence

  • Disassembled rpmalloc both ways on linux/arm64; with the tuning off the allocator is clean armv8-a, and the instructions above appear only with it on.
  • Built the image natively on an M3 Max and ran it — see the comment below for the result.
  • cargo fmt and cargo clippy clean on build.rs. (Clippy reports three pre-existing errors in lore-revision/src/{stage,state}.rs, untouched here and present on main.)

Note

Split out of #180, which touches the same two sections of DOCKER.md — whichever merges second will need a small conflict resolution.

.cargo/config.toml pins -C target-cpu=neoverse-512tvb for
aarch64-unknown-linux-gnu, so an arm64 image built from source SIGILLs on
anything short of Graviton3+. DOCKER.md worked around it by telling Apple
Silicon users to build linux/amd64 and emulate.

Assemble RUSTFLAGS in the Dockerfile instead. Overriding the pin needs
RUSTFLAGS specifically, not CARGO_TARGET_<triple>_RUSTFLAGS: the latter is
another source for the same config key and cargo joins config arrays, so
-C target-cpu would survive. ARM64_TARGET_CPU opts back into the tuning for
a Graviton deployment.

RUSTFLAGS does not reach a C compiler, and lore-base pins the same -mcpu
for its cc build of rpmalloc. That flag raises the architecture floor as
well as the tuning -- GCC 14 emits stlur (FEAT_LRCPC2, armv8.4) and ldapr
(armv8.3), undefined on Neoverse N1 parts such as Ampere Altra and
Graviton2. build.rs now takes LORE_ARM64_TARGET_CPU, defaulting to the
value it hardcoded, and the Dockerfile passes the choice through. Verified
by disassembling both builds: with the tuning off, the allocator is clean
armv8-a.

EXPOSE defaults to TCP, so it was not declaring the QUIC listener sharing
41337; docker run -P left it unreachable.

Signed-off-by: Aleksander Arruda <aleksander.arruda@xa.epicgames.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added documentation Improvements or additions to documentation area:server Server, provider integrations, telemetry area:core Core library and its interfaces (lib, C API); revision, storage, transport, protocol internals labels Sep 10, 2026
@aleksanderllada

Copy link
Copy Markdown
Contributor Author

Built and ran on an M3 Max, which is the case main documents as impossible:

$ docker build -f lore-server/Dockerfile -t loreserver:arm64-native .
$ docker image inspect loreserver:arm64-native --format '{{.Os}}/{{.Architecture}}'
linux/arm64
$ docker run --rm loreserver:arm64-native --version
loreserver 0.9.1-nightly

No --platform override, no emulation, exit 0.

Disassembled the shipped binary rather than trusting the flags:

stlur (armv8.4):  0
ldapr (armv8.3):  0
SVE:              0
casa/casal/ldadd: 2 each

Those last ones look like a hole in the baseline claim but are not. Every occurrence sits inside an __aarch64_* helper:

casa   <__aarch64_cas4_acq>
casal  <__aarch64_cas8_acq_rel>
ldadd  <__aarch64_ldadd8_relax>

That is rustc's outline-atomics, on by default for aarch64-unknown-linux-gnu: the helpers branch on __aarch64_have_lse_atomics at runtime and fall back to LL/SC, so they are safe on a CPU without LSE. The binary really is armv8.0-clean.

Not verified here: that --build-arg ARM64_TARGET_CPU=neoverse-512tvb produces a binary that faults on non-Graviton hardware. The rpmalloc disassembly in the description shows the C half taking the tuning, and the Rust half is the pre-existing .cargo/config.toml behaviour this works around.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core Core library and its interfaces (lib, C API); revision, storage, transport, protocol internals area:server Server, provider integrations, telemetry documentation Improvements or additions to documentation

Development

Successfully merging this pull request may close these issues.

1 participant