| feature | staging-workflow |
|---|---|
| start-date | 2018-03-05 |
| author | Vladimír Čunát (@vcunat) |
| co-authors | Frederik Rietdijk (@FRidh) |
| related-issues |
Define a new workflow for the staging branch that can better accommodate the
current and future influx of changes in order to deliver mass-rebuilds faster to
master. As part of this new workflow an additional branch, staging-next, shall
be introduced.
The current workflow cannot handle the high amount of mass-rebuilds that are
continuously delivered, resulting in long delays for these deliveries to reach
master. When a certain delivery causes failures, attempts are typically made to
fix these failures and stabilize staging so that the specific delivery can still
reach master.
Often it happens that during this period of stabilization other mass-rebuilds
are submitted, and it is not uncommon that these also introduce failures, thus
again increasing the time it takes for a delivery to reach master. This is
especially worrisome in case of security fixes that need to be delivered as soon
as possible.
There shall be the following branches:
masteris the main branch where all small deliveries go;stagingis branched frommasterand mass-rebuilds and other large deliveries go to this branch;staging-nextis branched fromstagingand only fixes to stabilize and security fixes shall be delivered to this branch. This branch shall be merged intomasterwhen deemed of sufficiently high quality.
Binary packages shall be build by Hydra for each of these branches. The following table gives an overview of the branches, the check interval in hours, amount of shares, and the jobset that they build.
| Branch | Interval | Shares | Jobset |
|---|---|---|---|
master |
4 | High | release.nix |
staging-next |
12 | Medium | release.nix |
staging |
6 | Medium | release-small.nix |
The check interval of staging-next is reduced from 24 hours (the current value
for staging) to 12 hours. This can be done because only stabilization fixes
shall be submitted and thus fewer rebuilds shall typically have to be performed.
The staging branch shall have a short interval of only 6 hours. This is because
of the relatively small jobset, and to obtain a higher resolution to detect any
troublesome deliveries.
A potential drawback of this new workflow is that the additional branch may be
considered complicated and/or more difficult to work with. However, for most
contributors the workflow will remain the same, that is, choose master or
staging depending on the number of rebuilds.
The current situation could be kept, however, that would not solve any of the issues mentioned in the "Motivation" section.
Instead of multiple branches only a single branch, say master, could be kept
for development. While this removes the issue of merge conflicts, it will result
in continuous mass-rebuilds on master, slowing down the delivery of binary
substitutes and thus development.
Reducing the size of the Hydra jobset would mean the iteration pace could be higher, but has the downside of testing fewer packages, and having fewer binary substitutes available.
The part about fewer binary substitutes could be partially mitigated by adding another slower larger jobset that wouldn't block the channel.
- The exact amount of shares, which is something that has the be found out.
- Document the new workflow;
- Create the new branch;
- Create a Hydra jobset for the new branch and adjust the existing jobs.