How should we handle parallelism in builds?

[kylewlacy]

For many packages, the build instructions will recommend running the build like make -j$(nproc), which will allow the build steps to run in parallel, up to the number of CPU cores on the host system. Running Make in parallel like this-- or other sources of parallelism in a build-- could introduce non-deterministic builds or flakey builds. The Reproducible Builds project even includes it in their list of "commandments of reproducible builds":

6. Thou shall only do one thing at a time or ensure races do no harm (parallelism)

...but this is very dependent on the build itself. If a package has a Makefile where all the dependencies are specified properly and where each step is itself reproducible, then adding parallelism shouldn't influence the final result. The benefit is the build itself will take only a fraction of the time: a 30 minute build could finish in less than a minute with enough CPU cores!

While we don't currently aim for making packages Reproducible today (by the "Reproducible Builds" definition), it'd be nice to keep the door open. Also, even ignoring Reproducibility, there are potential concerns around flakiness: a build that works with a parallelism of one could fail with some higher level of parallelism due to missing Makefile dependencies (or equivalent in some other build system)

So, we should have some standard guidance when authoring packages about if/when we should enable parallelism

[kylewlacy]

For a bit on the status quo:

• For the packages in std.tools() / std.toolchain(), I haven't enabled any parallelism. This was deliberate early on to avoid any unexpected issues from parallelism, but the toolchain has felt solid enough where it might be worth turning it on just to see what breaks. I'd probably lean towards leaving off parallelism still though
• For make (automake, etc) builds, we generally don't have parallelism enabled at all
• For the Linux build, it's currently set to build with make -j16 (linux/project.bri). I wrote this, and I did it to give it some much-needed parallelism but with a fixed amount of workers. It wouldn't be any faster on systems with less than 16 cores, but may still help to make sure the build is reliable regardless of how many cores the host has
• For other language build systems, we just do whatever the default is. For example, Cargo builds crates in parallel by default

[kylewlacy]

Also worth talking about the Brioche side a bit. Today, the linux_namespace sandboxing backend doesn't control the number of cores the build has access to: it doesn't use cgroups at all. It might be worth adding cgroups support purely for configuration reasons (e.g. prevent a build from keeping the CPU at 100%), but that probably intersects with how we should handle builds in brioche-packages.

Maybe it'd make sense to have a --parallelism flag (or equivalent env var or config field) to set the number of cgroups? Maybe the sandbox could expose that value directly to the build as an env var like $BRIOCHE_PARALLEL or similar then? Then, it'd make sense to use that as the standard for parallelism. The benefit of doing so over $(nproc) would be that we could change it even without using cgroups, so it'd affect the build even without any form of sandboxing. We could also clamp it to some minimum number (2 or 4?), so single-core environments still try to build with some parallelism as a way to try and catch flakiness.

[kylewlacy]

The nproc docs also say you can control the parallelism through the env vars $OMP_NUM_THREADS and $OMP_THREAD_LIMIT. Maybe starting with $(nproc) then tweaking it using env vars is the better option?

[jaudiger]

I was thinking to have a something similar to what we already have through the std.CURRENT_PLATFORM:

function getCurrentPlatform(): Platform {
  const { op_brioche_current_platform } = (globalThis as any).Deno.core.ops;

  if (op_brioche_current_platform === undefined) {
    return "x86_64-linux";
  }

  return op_brioche_current_platform();
}

export const CURRENT_PLATFORM = getCurrentPlatform();

We could have a function getting the number of CPU core / threads or whatever on Brioche side, this number would be exposed as a value through Deno, and we could then use it in the recipes. Of course, it would still give us some flexibility to disable parallelism or override the number of cores seen by introducing a new flag --parallelism as you mentioned. We could have something like that:

• Flag no defined: default value extracted from https://docs.rs/num_cpus/latest/num_cpus/fn.get.html or whatever else
• Flag is defined and value > 1: then we pass it
• Flag is defined and value == 1: also we pass it and it mimics a build without parallelism

I think in the CI, we could override the flag to always have a value greater or equal to 8 if the number of cores is less than 8.

But just to understand something, let's take the following example, what if the CI builds a package and published it with let's say std.CPU_CORE == 8 and I try to run the exact same recipe in my local environment with a value different, let's say 4, will the build be done again ? If I'm not mistaken, the answer is yes, isn't it ?

[kylewlacy]

But just to understand something, let's take the following example, what if the CI builds a package and published it with let's say std.CPU_CORE == 8 and I try to run the exact same recipe in my local environment with a value different, let's say 4, will the build be done again ? If I'm not mistaken, the answer is yes, isn't it ?

Basically yeah: say we added a value like std.CPU_CORE or whatever, which in JS was just a plain number value (meaning you could use console.log to see the value, add or subtract or do other math in JS with the value, etc.). If you passed that value into a build, then the build would only be cached for the exact same value of std.CPU_CORE, which would not be ideal...

A slight tweak we could do is to make std.CPU_CORE a "symbolic value" (to invent new terminology). Basically like how the value std.outputPath / $BRIOCHE_OUTPUT expands to a specific path at runtime, std.CPU_CORE could be the same way. That means if you tried to console.log it, you'd just see a weird object value instead of a number, and you couldn't do any math to it on the JS side, etc.-- the runtime just recognizes it and expands it to the core count right before running the actual process.

If it worked that way, you'd get a cache hit regardless of the core count: if you ran the process echo ${std.CPU_CORE} > $BRIOCHE_OUTPUT, it would be cached the first time you ran it. If you, say, upgraded your CPU from 8-core to 16-core and re-ran it, it would still be cached with the old value. But then again, is already kind of how things work today with the nproc command, which would have the same result for the cache.

[jaudiger]

I see, then I'll go for the straightforward approach you already used in socat recipe. I'll use the nproc command in the recipes using make commands. And set the environment variable OMP_NUM_THREADS when we need a number of threads greater than 8.