Support system package installation as a pre-run step

[liuliu]

This can be controversial, because it just opens another can of worms. But anyway, would like to open for discussion and may be I can learn something new or help to contribute the CI pipeline update.

Some niche packages, for example, my own: https://github.com/liuliu/swift-mujoco/blob/main/Package.swift#L27 has systemLibrary dependency that can be satisfied through brew / apt installation. This helps people who use library on the same platform they compile to use new capabilities.

However, this does limit cross-compilation effectiveness. And because you can add any packages, this is getting dangerously close to letting SPI run as a CI tool, which I guess the maintainers don't want that to be.

Here is the middle-ground suggestion: because SwiftPM supports this in a very limited fashion anyway, it seems reasonable to simply install through .systemLibrary's instruction and let it go wherever it takes. apt installation most of the time are very reasonable (all binary distributions, and the size is well controlled), brew is a bit tricky as some times it may install from source (???, not exactly sure about this assertion). But seems we can time / size limit the pre-run step to make sure it is reasonable.

[finestructure]

Thanks for opening that can of worms, @liuliu 😆

We currently have a workaround for this in place for Linux (see "Images for Linux" in this blog post) but it's not automatic.

For builds on Apple platforms this would be pretty much infeasible at the moment, because the runners are not running in throw-away virtual envs. I.e. if two builds were to be scheduled on the same runner installing incompatible or conflicting dependencies they would interfere. Leaving aside that they'd also leave their dependencies behind!

I think the only way to tackle this would be to use a dedicated virtualised macOS runner for packages the require dependencies that we can reset. This would allow us to continue with the rest of the setup unchanged. It's doable but quite a bit of work.

[liuliu]

Seems using virtualized macOS runner is the standard nowadays. Do you run into any limitations (perf?) and choose the current way? If there are any concerns and benchmarks you are trying to hit with, I can help to look into the virtualized runners. (After all, just Virtualization framework, right :D)

[finestructure]

At the time we first deployed the build system, virtualisation wasn't really an option for a number of reasons. But even if we had to do it again today, it wouldn't necessarily be our first choice.

For instance, we're essentially using a virtualised setup for our Linux builders where the builds run in docker containers. While it allows us to very easily build for various Swift versions, it comes with quite a bit of additional complexity around managing the containers themselves.

It's interesting and instructive how that advantage on Linux - all you need to do is change the docker image tag from swift:5.6 to swift:5.7 to build for a new version - doesn't exist even if you virtualise on macOS. You'd still need to deal with macOS and Xcode versions, it's just that you have to do in yet an additional layer on top of the OS you also need to manage.

Since we're not aiming to be a multi-tenant CI provider, keeping it all non-virtualised and single-tenant is a big help in keeping this manageable.

It would be possible in principle to extend our build system with a new platform flavour that runs virtualised and supports OS-level package dependencies. However, setting all that up isn't as easy as adding a virtualised runner (which isn't all that easy in itself). It'd also require significant instrumentation to a) figure out which packages to target to that runner and b) run set up and tear down tasks to clean up the environment for each build.

I don't have any stats how many packages are affected by this limitation. I suspect it's not many, probably less than 1%. That feels like a lot of initial work + subsequent maintenance overhead for a small audience.

Given that, I think an easier fix would be to come up with some of tie-in to an existing CI that these packages are running. I'm not saying it's possible or easy or how it would work exactly - but I think a better approach to us building even more of a CI system would be to allow certain packages to report build results to us in place of our runners.

[liuliu]

Thanks for the detailed answer. I think that I need to study your setup to come up with a more appropriate suggestion. My original thought is that using macos vm would make it more manageable, maybe with some performance related issues, not less. If it makes things less manageable, it is obviously a no go.

Regarding how to figure out which package, I still stand by my original suggestion to just infer that from Package.swift, this is restricted, middle-of-a-road choice.

Reporting back sounds great, but I think that lost authority of the SPI, which basically promises if I can build, you can too.

Again, thanks for the answer, if you have any pointers to your current setup such that I can study more to come up with a suggestion that makes using vm a more manageable choice than status-quo.

[CamJN]

As someone with a package that doesn't build if a library isn't installed, I'm interested in this being solved. I can provide the library as a binary & header, or an installer pkg, or even a homebrew formula, though not one in the homebrew-core tap.

[daveverwer]

Out of interest, which package is it and what are the pre-requisites?

[daveverwer]

As someone with a package that doesn't build if a library isn't installed, I'm interested in this being solved. I can provide the library as a binary & header, or an installer pkg, or even a homebrew formula, though not one in the homebrew-core tap.

Thanks for the additional perspective @CamJN

We are considering a change to our build system right now which may make this more feasible. I'm not at the stage where I can promise anything yet, and if/when we go ahead with it, we also will need time to get the new system stable before supporting custom environments like this, but we are doing active work in this area, and so all hope is not lost!