Spawn recipes within recipes (sccache-like builds)

[kylewlacy]

In a normal C-based project, each .c file can be compiled separately into a .o file, then all the .o files can be linked into a final binary sccache is a tool that hooks this process, so the compilation of each .c file can be cached remotely, which can speed up build times massively if a build is mostly unchanged.

Basically, sccache works using a custom cc wrapper. In short, when it's called to build a .o file, it first preprocesses the input .c file (e.g. using gcc -E), which then acts as the cache key. It checks to see if a build for the preprocessed input already exists. If it already exist in the cache, it fetches the result and writes it to disk; if it doesn't exist, it calls the C compiler for real to build the .o file and saves the result to the cache. sccache also supports Rust by wrapping rustc, which follows a pretty similar flow.

We already have our own cc wrapper. But there's no way for this to interact with the Brioche cache today. But if we had a way for our cc wrapper to spawn another recipe, that would let us automatically break a build up into smaller recipes for faster rebuilds-- basically emulating how sccache works, but using the Brioche cache. Also, if/when we support remote builders (#241), that could even let us distribute a build across many machines.

---

To support this, the main thing we need is the ability for a running process to submit a recipe to the main Brioche process using some form of IPC-- the recipe would then be baked like any other, and the recipe's output would also need to be sent back to the process somehow.

Also, we'll need to associate the inner recipe with the outer recipe. That way, it can be synced properly, shown in the UI properly, and debugged properly. A few possibly-relevant issues: #318, #339, #340

Add IPC socket to sandbox

The simplest way for a running process to communicate to the main Brioche process would be to create a Unix Domain Socket within the process's filesystem.

On the Brioche side, we'll add a new variant for ProcessTemplateComponent (a.k.a a "symbol"), which will expand to the path the process can connect to in order to send these messages. Something like BriocheHostSocket, which then expands to the UDS path when used within a process template.

(I'm also split on whether the socket should just be a plain filepath, or if it should be a URL! It might make sense to use a URL in case we want to also support other IPC mechanisms where UDS isn't a (good) option, or even TCP/HTTP URLs to allow for building remotely. I'm kind of leaning towards just using a path to keep things simple for now)

Implement initial IPC protocol

Once the socket is set up, the Brioche host process will exchanges messages with child processes like this (not a real protocol, just pseudocode):

child>
{message: "bake-recipe", output: "/tmp/...", recipe: {type: "process", command: "gcc", ...}}

brioche>
{message: "bake-finished", status: "succeeded"}

(There's many, many options for both the communication protocol and the message serialization format!)

When the Brioche host receives a bake-recipe message, it'll bake the recipe like normal (or fetch from the cache if possible). Once finished, it'll write the output back into the sandbox (we may want to make this optional, instead sending the output contents over IPC too... but it'll make the child-side a lot easier if the host can just put the process directly in the sandbox)

We'll also want to place a few restrictions on spawned process recipes:

• The new process recipe can't have more unsafe settings than the child
• The new process recipe can only include artifacts that were also included in the child, or created within the child

Also, to handle artifact messages, I think we'll need at least 2 more messages:

• get-inputs: List the input artifact hashes and their paths included in the child. This will be used to find dependencies for the new process, e.g. gcc
• create-artifact: Takes a path, returns an artifact hash with its contents. This will be used to create inputs for the new process, e.g. the (preprocessed) .c file to build

(std) Use an env var for IPC socket by convention

For most process "symbol" inputs, we choose an env var that's conventionally set for the symbol, e.g, $BRIOCHE_OUTPUT for OutputPath.

For the new host socket symbol, we'll pick an env var and use it for the IPC path by convention, e.g. $BRIOCHE_HOST_SOCKET_PATH.

In std, we have 2 options:

1. Set this env var by default in std.process(), opt-out by overriding it
2. Don't set the env var by default, explicitly opt-in to setting it (i.e. with a method like std.Process.withHostSocket() or something)

Personally, I'm leaning towards (1)

(runtime-utils) Update cc wrapper to (optionally) spawn process recipes

As mentioned, we already have a cc wrapper in the runtime-utils project. The existing wrapper will be updated so it can spawn processes remotely, essentially following these steps:

1. If -c flag is not included, just build locally (we only care about caching .o compilation, not linking)
2. If $BRIOCHE_HOST_SOCKET_PATH is not set, just build locally
3. If spawning recipes is disabled based on the $BRIOCHE_CC_SPAWN_RECIPES env var (open to bikeshedding), just build locally
4. Find toolchain artifact using $PATH and get-inputs. Bail if not found.
5. Preprocess the input file(s) using $CC -E as needed
6. Create artifacts for (preprocessed) input files using create-artifact
7. Spawn a process to call $CC for each input artifact using bake-recipe
8. Wait for all bakes to finish successfully and return

I think it's also an open question on if we want this to be the default behavior or not. I'm leaning towards leaving it disabled, at least initially. Realistically, we might even set the default based on our needs for brioche-packages, and it'll take some work to figure out if this whole thing even speeds up brioche-packages builds at all! It could turn out to be a net-negative if most package updates end up causing lots of cache misses. Plus, it'll be a complete wash if the compiler changes (e.g. for Rust, we should expect to see full rebuilds every 6 weeks, since that's how often they do releases).