Description
See ocaml/ocaml.org#1169 for the overall tracking issue.
The current CI running on the opam repository is one based on datakit-ci (source code: https://github.com/avsm/mirage-ci). It is the infrastructure that listens for GitHub PRs and runs the custom checks required for every package accepted to the opam-repository.
Over the past few years, the datakit project has been retired upstream, and so we began implementing a new generation of infrastructure that could cope with the workflow needs of opam. In addition to testing the existing spectrum of Linux distributions, we also need to test macOS, Windows and the BSDs, as well as multiple architectures (specifically arm32/64, ppc64 and riscv). There is also a need to incorporate custom workflows for the next generation of security features in opam, for example conex software supply chain security.
In order to support this and retire the current datakit-driven infrastructure, @talex5 has been leading the development of ocurrent. It is designed to be configured using OCaml (see this for some detail), and has support for triggering work and doing deployments. The most current prominent example of using ocurrent to build a CI is explained in the "zero configuration CI for OCaml" talk at this year's OCaml Workshop.
This ticket tracks the progress of migration from datakit-ci to ocurrent. Before we do so, we need to document the various pieces on ocaml.org:
- base-image-builder: this generates and pushes the Docker container images used by (among other things) the https://github.com/ocaml/ocaml-ci-scripts. Information on the ocaml/infrastructure wiki needs to be moved over to ocaml.org, such as the available containers and build hosts. The containers information is in particular important to signpost changes for as our community users will have a dependency on them in their CIs. One problem are is that the ocaml/opam2 containers cannot currently be easily updated, so we need to do a post on discuss.ocaml.org to explain why and what will change.
- ocluster: a live cluster scheduler that can schedule tasks among the VM compute cluster. This is largely internal but exposes a Capnp interface, so could be used for custom build tasks (e.g. Incubate docs.ocaml.org #1171 for docs building).
- opam-repo-ci: prototype exists at https://github.com/ocurrent/opam-repo-ci and is being ported to ocluster at the moment to facilitate the migration.
From a user perspective, what we need on ocaml.org is an explanation of the tests that are conducted when a package is submitted to opam repository, and which are automated and which are manual by the repository team. There are a number of FAQs that the opam-repo team have fielded over the years which would benefit from being documented more clearly directly in ocaml.org.