Cross-compilation support

[kylewlacy]

Context / other discussions:

• May 2025 Project Update: "Work on cross-platform support"
• Zulip: "Differentiation between runtime and buildtime dependencies?"

At the time of writing, Brioche supports native builds for x86_64-linux and aarch64-linux (nightly-only), but we don't have any explicit support for cross-compilation today. To support cross-compilation, we will fundamentally need a few new things:

1. Separation between "build dependencies vs. runtime dependencies" (a.k.a. "host dependencies vs. target dependencies").
2. A way to set the target platform for a build. Something like a --target flag, e.g.: brioche build --target aarch64-linux.
3. A way to get the target platform for a particular recipe. This can differ from what's set in (2) because a host dependency will need to be built for the host platform.
4. A cross-compiling toolchain (in brioche-packages).

Beyond that, we have a lot of freedom to maneuver, but there are a lots of big and little design decisions and trade-offs that need to be made!

[kylewlacy]

To reiterate a bit from the May Project Update, my big design idea is to introduce the concept of "variables" at the recipe level. As a basic example:

export default function curl() {
  return std.runBash`
    # Build curl from source
    ./configure && make && make install
  `
    .workDir(source)
    .dependencies(std.toolchain, openssl)
    .hostPlatform(std.variable("hostPlatform"))
    .targetPlatform(std.variable("targetPlatform"));
}

The std.variable() function would return a "placeholder" we'd substitute when evaluating recipes-- basically a form of dynamic binding. That would let us "rebind" the variable contextually, like this example also from the May Project Update:

export default function () {
  // Build curl for x86-64 Linux
  const curlX64 = curl().setVariable({ targetPlatform: "x86_64-linux" });

  // Build curl for aarch64 Linux
  const curlArm = curl().setVariable({ targetPlatform: "aarch64-linux" });

  // Combine into a directory
  const curls = std.directory({
    "curl-x86-64-linux": curlX64,
    "curl-aarch64-linux": curlArm,
  });

  // Create a tarfile for the directory
  return createTarfile(curls);
}

This is maybe a niche example. Most packages probably wouldn't ever use setVariable (directly), but it would be used for "host / target" or "build-time / runtime" dependencies: host dependencies would need to be built for the host. For example, Bash normally just needs to run on the host platform, so somewhere there'd be a recipe like std.bash().setVariable({ targetPlatform: std.variable("hostPlatform") }), which "pivots" the recipe so bash can run on host platform at build-time

[kylewlacy]

From drylippedurchin on Zulip:

It would also make sense to have an ability to limit
cross-compilation targets in project, since not every platform and os can
be supported by every app (e.g. windows dependant app, can't be compiled for
linux, or say the app needs to hand-written assembly for some of its
functionality, or some magical linker scripts or whatever else that may limit
the options), so i should be able to say that this project can be built only on
windows under arm,aarch64 and x86-64 and be able to write separate ways
to handle different platform\os combos (since there can be essential
workarounds)

And definitely 👍 that we'll need to support this! Wanted to put a top-level comment for this because there's a few related questions in this space:

• Should we mark platform support at the project level or recipe level (or both)?
  • I think we'll at least want this info at the project-level for brioche-packages, see also Add project metadata for platform compatibility #268
• Should we mark cross-compilation support at the project level or recipe level (or both)?
  • I think this makes more sense at the recipe level. Builds can involve multiple recipes, and it wouldn't surprise me if, during a single build, there were some recipes that supported cross-compilation and some that didn't. Long-term, I think recipes that don't work on your platform could either be emulated (e.g. qemu) or run remotely. Having finer-grained control means we can e.g. emulate as little as possible.
• How should we handle "platform agnostic" builds? e.g. building a static site, where the "target platform" should be meaningless.

[MrDullDev]

How should we handle "platform agnostic" builds? e.g. building a static site,
where the "target platform" should be meaningless.

I think we can just add a dummy platform to accomodate that e.g. something like
platform-agnostic or none. Or we can just separate them out in their own
meta-platforms in a more specific way. Like jre platform for java bytecode,
web for web applications, config for text config files, etc.

[MrDullDev]

So after reading about variable idea a few times over, i finnaly got what it
is supposed to be and kinda like it. So this is a summarisation of my
understanding for my future self and other people that may be a little confused
about the idea:

The std.variable thing is just supposed to fetch the variable by the name
that is the argument, e.g. so std.variable("MyAwesomeVariable") returns the
value of a variable by the name of MyAwesomeVariable. The idea being that
it gets substituted by the brioche when recipe is being backed or by some
placeholder. To elaborate on that concept std.variable("hostPlatform") from
original example would return something like say gnu platform triplet (e.g.
aarch64-linux-gnu) or some pre-defined structure with the same info
or something like that.

I like this idea because it allows for cross compilation logic to be
implemented in recipe without weird hacks(e.g. some precompiller magic or just
some general macro substitution) and solves the problem of not being able to
access the brioche arguments or other specific things (e.g. like in our current
predicament host and target platforms, or some other thing in the future if
the need ever arises). So we can organically implement default cross
compilation for say rust cargo projects by just adding logic to parse the
that info from std.variable("hostPlatform") and
std.variable("targetPlatform").

[MrDullDev]

So after a bit of thinking, passing a raw gnu triplet(which is more of a
quaduplet if you really think about it) is kinda a bad idea for a few reasons:

• GNU syntax has no formal definition(so there quite a few quirks here and
  there)

• GNU syntaxa allows quite a bit of ambiguity(e.g. the aarch64-linux-gnu
  that i gave as an example earlier is actually not the gnu triplet in
  pure sense of the word, since it is in fact a quadruplet in disguise and
  full form of it is aarch64-unknown-linux-gnu where aarch64 - cpu
  arhitecture, unknown - manufacturer, linux - kernel, gnu - os/platform,
  but more like the target libc), so parsing it once should be a priority,
  since its actually hard to parse

• Certain toolchains/build-tools might be expecting a certain flavor of GNU triplet(due to the absence of formal definition, aka 1 reason in this list),
  so by passing it raw like that we might encorage the behaviour of just
  passing it as-is to the toolchain, which can lead to unexpected problems

Which leads me to the following points:

• I think that we should decide on some sort of type that would describe the
  platform in uniform way, that should be returned by the concerned
  variables.

• The type include the same info as the GNU triplet at a minimum, and more,
  so things like: CPU architecture, Kernel, Libc-flavor, ABI(if aplicable),
  compiler-framework(aka GCC, LLVM or whatever, if aplicable),
  executable-container(aka ELF, bFLT, a.out,PE, etc. if aplicable)
  compiler-flags(maybe in some sort of abstraction, if aplicable),
  linker-flags(maybe in some sort of abstraction, if aplicable)

• We probably should write a few platform-string generators to construct
  those funky things

My justification for including compiler flags, and linker flags is mostly about
enforcing some weird things like compiling for system with no FPU(aka no
hardware way to work with floating point values) or say for systems with no
MMU and some things like that) . As for other extended options its mostly
about future-proof customizability and niceties(i mean compile flags kinda
fall in that category). I mean i would love to have a hassle free way to do
weird things like running PE linux executables for my entiry stack or
directly comparing the perfomance of software stack build with llvm vs gnu,
and all those funky things.

It may also be nice to group compiler-flags by language-compiler framework,
so things like c-gcc and c-llvm flags, go-gcc,go-gc and go-tinygo,
etc. that way we can catch the whole group of mistakes and it will synargise
nicely with platform type having compile-framework. We can also separate
them out all-together, and make a subsystem of their own for compiler flags
and linker flags. That way we can separate out global flags, for all
aplicable recipes/projects, project/recipe-wise recursive flags(aka
flags that get recursively aplied to all project/recipe dependencies) and
local flags(aka that are aplied only to the recipes/project where they
were defined)

Another point i want to bring up is a way to override certain platform
things in certain dependency recipes/projects. Like say i want to compile
everything but curl with gcc and compile curl specifically with llvm or
I want to compile everything with -O3 but the curl that i want specifically
with -O0

[kylewlacy]

@MrDullDev yeah there's a lot of nuance to unpack when it comes to exactly what features / flags to pass to a compiler 😅

I'm kinda hoping we could keep the hostPlatform and targetPlatform variables really simple: x86_64-linux, aarch64-macos, etc. (basically like $GOARCH and $GOOS from Go, and the Platform enum we already use internally). I think we'd have more options for target for weird cases you mentioned (probably like x86_64-uefi for UEFI PE executables, and a bunch of arm*-baremetal ones for the wide range of ARM microcontrollers). Then, I was thinking we could add a separate toolchain variable, which would basically cover the C compiler, linker, libc, and the ABI.

Maybe this idea is too naive...? But I think it might work for Brioche because of our packed executables. Normally, users need to care about the full target triple since, for example, a curl binary built targeting Ubuntu/Fedora/another GNU-based distro (x86_64-unknown-linux-gnu) won't run on Alpine (x86_64-unknown-linux-musl) due to a different dynamic linker, libc, and ABI requirements. But for Brioche, since our curl package gets bundled with its libc and dynamic linker, it'll run fine on Ubuntu, Alpine, NixOS, or basically any distro¹. It could still matter if someone wants to link against the libcurl from Brioche's curl, but I see that as a more minor point, since you could solve that problem by just making sure you use the Brioche toolchain when linking with libcurl (I feel like compilers/linkers/toolchains across any two package ecosystems could get dicey for similar reasons!)

And I think we could have separate variables beyond hostPlatform, targetPlatform, and toolchain as the need arises, but not sure how fine-grained that would get yet...

---

For more fine-grained per-package control, I guess at the highest-level, I'm thinking that most folks won't care about overriding most flags for most packages. IMO Nix's override feature is really slick and it's definitely one of the trade-offs for using TypeScript instead of NixLang for Brioche: in exchange, I'm hoping using TypeScript makes things simpler and more approachable overall! We'll probably never reach the same level of flexibility that Nix's overrides give, but I can think of 3 different approaches, which make sense in different contexts (ordered from simplest to most complex):

1. Function arguments. Every package has an export default function for the main package recipe which can be called without any arguments. But we can add optional arguments to any package as we see fit (we do this today in python to change the version, and in a few other packages too)
2. Forking (and git dependencies). For direct dependencies, it's not too hard to just fork the dependency and manually edit the package manually. Today we only support registry dependencies and path dependencies, but if we allowed git dependencies, that would mean you could directly fork the brioche-packages repo wholesale, or just manually create a separate repo with one or a few packages. This of course has the downside that you'd need to maintain the fork and keep it in-sync with the upstream.
3. Dependency overrides. Package managers like Cargo, NPM, etc. usually have ways to override / patch dependencies, including transitive ones, e.g. [patch] in Cargo.toml. We could offer a similar mechanism. It would take some creative thinking, but I feel like we could turn this into a pretty powerful tool.

Taking your -O0 example for curl:

1. Function arguments: we could add an optimizationLevel argument to curl, or maybe an arbitrary cflags argument. (I would probably push back against this, since this seems pretty niche)
2. Forking: of course, just forking or copy/pasting the curl recipe and adding the appropriate $CFLAGS setting (or whatever). (Not a super fun option and could obviously be painful to do for a long-term change...)
3. Dependency overrides: this actually opens several doors. If curl is a transitive dependency, then you could just use (2) with an override. Or, you could do something silly like override the std package used by curl, which would give you full power to set whatever env vars you want. That feels like extreme overkill though and very invasive...

None of these feel super compelling, so I don't really have a slam dunk idea for how you could set just -O0 scoped to a single package...

Footnotes

1. There are still some "minimum requirements" for e.g. our version of curl. Namely, the system needs to support the ISA and kernel version we're targeting, e.g. x86-64-v3 and Linux >=v4.14, respectively (I think). So I guess, if we wanted to be super explicit about our support, a more "correct" platform name would be something like x86_64_v3-linux_4.14? ↩

[kylewlacy]

Thinking out loud a bit more: recipes in Brioche today are very static overall, and the "variables" idea basically makes recipes a bit more expressive. But bigger picture, I feel like there's still a lot of untapped potential around recipes. The hard part IMO isn't just making recipes more powerful, but coming up with easy-to-use interfaces to expose that power.

...anyway, recipes have enough metadata where we could say "insert the $CFLAGS env var when building curl but not its dependencies", although I think the hardest part is figuring out a sensible interface to express that from TypeScript...