Composable / fluent commands

[kylewlacy]

This concept came up during a PR review recently

We basically have 2 different ways to run a single command as a build step today:

(1) Using std.runBash to run a shell script from a string (or runNushell, etc.)

std.runBash`
  gcc hello.c -o "$BRIOCHE_OUTPUT/bin/hello"
`
  .workDir(source)
  .dependencies(std.toolchain)
  .outputScaffold(std.directory({ bin: std.directory() }))
  // ... other options ...

(2) Using std.process to make a process recipe "manually"

std.process({
  command: "gcc",
  args: ["hello.c", "-o", std.tpl`${std.outputPath}/bin/hello`],
  workDir: source,
  dependencies: [std.toolchain],
  outputScaffold: std.directory({ bin: std.directory() }),
  // ... other options ...
});

I think both utilities are okay, but I think we can do better. A few thoughts:

• std.process is a more direct building block. We just run whatever process we're given without a shell in the way. No need to worry about escaping inputs, since we don't do shell expansion
• std.runBash I think (hope!) is more intuitive for a beginner. It looks pretty close to how you'd compile a C project "by hand" (hoepfully, it'd be pretty easy to copy/paste existing shell commands and convert them to a Brioche build without much effort)
• std.process doesn't require bash during the build, which could help eliminate un-needed dependencies... in theory, at least!
• std.process can only run a single process! If you need multiple processes within the same recipe / sandbox, you'll need another tool: a shell (or something else to run multiple commands in sequence)
• In either case, outputScaffold looks kinda ugly / complicated
• In either case, "chaining" steps together is a bit of a pain. Basically, you need to take a recipe, then use .pipe(...) on it to use it as the outputScaffold for the next step.

---

One idea I had thrown out (in the PR review mentioned before) would be a utility to that abstracts over .outputScaffold(), and @jaudiger suggested that we could even generalize it further and let it take an arbitrary path. Something like .mkdir(std.tpl'${outputPath}/bin') to create an output path, or .mkdir(std.tpl'${workDir}/build') to create a path in the work dir.

Taking this idea to the extreme, I think it'd be cool if we had a more composable way to run commands. Ideally, we could:

• Support chaining multiple commands. We should support both chaining multiple commands within a single recipe and chaining multiple recipes together.
• Support more ergonomic "setup" steps. Translating from shell mkdir / cd commands to .outputScaffold() / .workDir() / .currentDir() is a bit annoying today.
• Allow using shell-specific features when needed, but default to using process recipes directly without a shell where possible

[kylewlacy]

I think looking at Dockerfiles is an interesting case study. Don't get me wrong: I think Dockerfiles have a lot of issues and make doing things the right way pretty painful (and I could go on about my complaints about Docker itself)... but Dockerfiles do effectively solve the problem I'm outlining. I think you could even take a Dockerfile and translate it to a set of Brioche recipes pretty directly.

Actually, here's a pretty basic example:

RUN echo foo > foo.txt
RUN echo bar > bar.txt

...and here's a basically-equivalent Brioche recipe today:

std.runBash`
  echo foo > foo.txt
`
  .currentDir(std.outputPath)
  .outputScaffold(std.directory())
  .toDirectory()
  .pipe((recipe) => {
    std.runBash`
      echo bar > bar.txt
    `
      .currentDir(std.outputPath)
      .outputScaffold(recipe)
      .toDirectory()
  })

This isn't so much to say we should emulate Docker's model, but it shows that they're basically on the opposite end of the spectrum from how we do things.

---

Along the same lines, Earthly Earthfiles are a sort of evolved form of Dockerfiles. I've really only skimmed the basics of Earhtly, so there might be some interesting ideas to learn from there!

[jaudiger]

Support chaining multiple commands. We should support both chaining multiple commands within a single recipe and chaining multiple recipes together.

100% agree with that, we should better support chaining recipes and processes. I kind of like what we did here and here. It would be good if we could expand this pattern.

Support more ergonomic "setup" steps. Translating from shell mkdir / cd commands to .outputScaffold() / .workDir() / .currentDir() is a bit annoying today.

Based on the idea of adding a mkdir method (.mkdir(std.tpl'${outputPath}/bin'), .mkdir(std.tpl'${workDir}/build'), we could have a cd method that would be similar: .cd(std.tpl'${workDir}') to go inside the work directory or .cd(std.tpl'${outputPath}') to go inside the output directory. With both mkdir and cd commands, we could replace .outputScaffold() / .workDir() / .currentDir()

---

The main issue with adding these commands and chaining things is to enforce proper ordering during the execution of the commands. Basically to take an example, when calling .mkdir(std.tpl'${outputPath}/bin'), it should be clear (by just reading the recipe source code) that the directory will be created before running the bash script or process. At the end, it could look like that (just an idea to brainstorm here):

cd(source)
  .mkdir(std.tpl'${outputPath}/bin')
  .runBash({ script: `
    gcc hello.c -o "$BRIOCHE_OUTPUT/bin/hello"
  `, dependencies: std.toolchain })

If I follow my idea that the order is important, since each method will setup the environment step by step, I moved .dependencies() in .bash() to enforce that the dependencies are only used in the bash script. Having a separate .dependencies() doesn't make a lot of sense, since we can't now the scope of .dependencies(). Unless we had... a scope method:

cd(source)
  .mkdir(std.tpl'${outputPath}/bin')
  .scope(dependencies: [std.toolchain], (recipe) => {
     std.runBash`
      gcc hello.c -o "$BRIOCHE_OUTPUT/bin/hello"
    `
  })