Examples of enhancing crane for larg and compex projects

[dpc]

Hi @ipetkov

In flakebox I'm going to work on bringing some abstractions and functionality that would make it easier to replicate the really nice dev shell / CI setup we have in Fedimint. And I want everyone (Rust + Nix project) to learn love and use it.

One thing I had to add is "enhanced crane":

https://github.com/rustshop/flakebox/blob/233129a439d5f5eb8cb3ba4e996355d71483ac0e/lib/crane/enhance.nix#L1

As you can see I've added the the "commonArgs" support I described before
. I named it just args here.

This allows me to build all sorts of abstractions like "map over all relevant profiles" or "map over some cross compilation targets".

With a bit of these things composed I can now easily do things like:

> uname -a
Linux mutex 6.5.1 #1-NixOS SMP PREEMPT_DYNAMIC Sat Sep  2 07:13:30 UTC 2023 x86_64 GNU/Linux
> nix build -L --show-trace .#aarch64-linux.release.flakebox
> ls -alh result/bin/
total 13M
dr-xr-xr-x 2 root root 4.0K Dec 31  1969 .
dr-xr-xr-x 3 root root 4.0K Dec 31  1969 ..
-r-xr-xr-x 2 root root  13M Dec 31  1969 flakebox
> file result/bin/flakebox 
result/bin/flakebox: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /nix/store/f1c3qv0miwni8hx6jm1ym30vfh423nwi-glibc-aarch64-unknown-linux-gnu-2.37-8/lib/ld-linux-aarch64.so.1, for GNU/Linux 3.10.0, with debug_info, not stripped

which is very handy e.g. for running the CI with some build-time-optimized ci profile while retaining the ability to generate release binaries, etc.

I also added the buildWorkspaceDepsOnly, buildWorkspace, and buildCommand. These are meant to handle workflows for larger projects, like described in #396 .

I just wanted to show you what I'm aiming at. Most projects nowadays just have one derivation in release build and waste time and space, but not mind it too much. This really breaks down in a compilation-heavy language like Rust very quick as the size of the project grows.

I wish some of this was built-in into crane. I don't think it's a lot of code or even complexity, and it would really help the users if they didn't have to be aware that Flakebox's has a hacked craneLib version and some of this stuff will not work with plan craneLib, or will not match the documentation. I would rather point users of fakebox to crane's documentation for details.

I would rather focus in Fakebox on other stuff (loops over profiles, over toolchains, cross-compilation, etc.). Stuff that is out of scope for craneLib as it integrates it with other pieces.

I'm not planning to ask for more complexity later. I think "common args" and some some basic extra commands are all that is really needed.

[ipetkov]

One thing I've been thinking about is introducing something like craneLib.contrib which could have a lot more opinionated/preconfigured options for things. The separate contrib namespace would also highlight the nature of those helpers (e.g. they might do things more differently than one would expect with plain old mkDerivation, they might not be as configurable as the rest of crane)

That might be a good middle ground for introducing extra things like that!

[dpc]

@ipetkov I'm open for using solutions like that, though I wish to minimize complexity for the user and maintaince.

BTW. Just want to say that I've been studying crane code a lot and I'm learning and copying a lot of things it does. Your work is much appreciated. Just yesterday I added checks mimicking what crane is doing.

At the core, I think you want craneLib to be exactly like pkgs but with extra methods. Flat namespace. I think I get the idea, as it keeps things simple and similar to what nixpgs is offering.

I got stuck on this idea of craneLib working more like it's own thing. Sort of an object/scope for handling Cargo project. It grew on me when working on more and more complicated setups, and commonArgs // becoming rather unwieldy. In larger project I can have 40 or more craneLib-using derivations and I want them to look as compact as possible, in particular so they are easier to understand and copy&paste by not-yet-proficient members of my teams.

I've been thinking about a what is the best road here.

Right now I have a small wrapper + a fork with 3 patches. Which is just a result of me building a PoC, and I might need to make it cleaner one way or another. I would prefer to mostly get rid of this whole file somehow.

One approach for me is just to go full separate library on top of crane. Instead of intercepting mkCargoDerivation, just build a whole new layer, exposing only calls that are support, instead of trying to infer all existing functions and hope they work. But that ultimately should go along with having own name, documentation and not just "refer to crane documentation" everywhere. And maintaining docs is an ungrateful chore (at least to me).

Another one is keeping things as they are, trying to merge into crane everything that seems to fit, and explain to the user that "it's a craneLib, but with extra things".

Another is having a fork that is explicitly "the original with extra things on top".

I'm not sure craneLib.contrib gives me any further options, because that `craneLib - being own scope" is not just some extra methods. It's a fundamentally different way of working with crane, that puts these "common args" at the very root.

[ipetkov]

I'm not sure craneLib.contrib gives me any further options, because that `craneLib - being own scope" is not just some extra methods. It's a fundamentally different way of working with crane, that puts these "common args" at the very root.

FWIW I don't think this is actually a limitation here. Semantically you could do something like:

let
  craneLib = ...; # this is the `craneLib` we have today
  craneLibOpinionated = craneLib.overrideScope' (final: prev: {
    inherit (prev) craneUtils;
    mkCargoDerivation = ...; # Patched `mkCargoDerivation` to do extra things
  });
in
lib.attrsets.unionOfDisjoint craneLib {
  contrib.opinionated = craneLib.opinionated
}

Then you could entirely use craneLib.contrib.opinionated.buildPackage or anything else!

---

FWIW though feel free to experiment in a separate repo if you'd like to "go faster" and we can fold the good ideas in behind craneLib.contrib. Or alternatively if it turns out to be too different/unwieldy to add it that way we can always "promote" the repo as a sister-project to crane itself, put them both in a github org, etc.

[dpc]

That might work.