An improved workflow for maintaining Salt #96
Conversation
meaksh
force-pushed
the
master-new-maintaining-workflow-for-salt
branch
from
00367fd to
a8616ca
Compare
|
|
||
| As mentioned this is now at `pkg/suse/salt.changes` in `openSUSE/salt` GitHub repo. | ||
|
|
||
| When creating a PR to `openSUSE/salt` the user must also include the corresponding changes to the spec file, that can be generated as usual with `osc vc`. |
There was a problem hiding this comment.
Not sure if I get this point right. Maybe it says spec file but the actual meaning is different, not sure, could you please clarify it?
There was a problem hiding this comment.
Woops, sorry. I meant "changelog" file 😄
Let me fix this
There was a problem hiding this comment.
Do I understand it right, that in this case we will add changelog entries manually to the changelog file or manually but with osc vc, still not fully clear here. With the osc services there is way to use commit messages as a changlog items, don't we want to use it this way?
There was a problem hiding this comment.
I'm proposing here to manually add the changelog entry to salt.changes by using the osc vc command directly in your Git tree, so the generated changelog entry (and header) can be included together with the code changes in the PR to openSUSE/salt repo.
This way a single PR to openSUSE/salt repo could contain all possible different bits: code changes, specfile changes, changelog entries, artifacts changes.
See this example PR: https://github.com/meaksh/salt/pull/10/files
Now that you mentioned the "osc services" for the changelog, I realized that the current proposal is not covering the fact that we need to maintain different changelogs depending on the targeted codestreams (as they are not aligned).
I'll have some thoughts and clarify this on the RFC text.
There was a problem hiding this comment.
I've added more text to the RFC on how to deal with different maintained changelogs
There was a problem hiding this comment.
We probably need a script that's a bit smarter and able to edit all the changelogs at once (e.g. to add a bsc number to the current entry). I have a POC, it's not too hard to have that and all the changelogs in the repo.
|
|
||
| When creating a PR to `openSUSE/salt` the user must also include the corresponding changes to the changes file, that can be generated as usual with `osc vc`. | ||
|
|
||
| Similarly to the main Uyuni repository, we should add a GitHub action to warn the user in case no changelog entry is added in the PR. |
There was a problem hiding this comment.
Are we also merging the changelog like we do in Uyuni or do we need to resolve merge conflicts manually?
There was a problem hiding this comment.
This should be now clarified on the RFC text
There was a problem hiding this comment.
I can't find it, can you help me out with the line number or a link? With the list of "backported PRs" in the spec file, we might hit this issue in two places.
| * package building according to PR branch. | ||
| * branched and removed automatically from `systemsmanagement:saltstack:github/salt` by OBS workflow. | ||
|
|
||
| The same OBS structure will apply to all our OBS targets: `products`, `products:testing` and `products:next`. |
There was a problem hiding this comment.
Can you explain how the structure applies to these projects concretely? I would have thought products:testing or products:next don't need a separate github subproject.
There was a problem hiding this comment.
Hmmm, actually for products we don't really needed, as we just copypac whatever is in products:testing to products.
But for products:testing and products:next I will also consider the github structure, to be able to have different Salt versions if necessary ensuring those packages are also ready to be consumed (even if those are never be directly released) but we would prevent enabled services can run unexpectely on targets that are linked to products:testing and products:next (like i.a. Uyuni:Master or D:G:M:*)
| <param name="scm">git</param> | ||
| <param name="versionformat">@PARENT_TAG@</param> | ||
| <param name="versionrewrite-pattern">v(.*)</param> | ||
| <param name="revision">openSUSE/devel/master</param> |
There was a problem hiding this comment.
I was not aware we used an openSUSE/devel/master branch. Is this a new branch we will use?
There was a problem hiding this comment.
I used openSUSE/devel/master here just as an example, it should point to the eventual openSUSE/release/3008.x branch. I'll fix this on the RFC text.
Currently openSUSE/devel/master is just the devel branch I created with upstream master branch + our patches partially rebased on top (excluding patches to extensions).
|
|
||
| ### Salt Extensions | ||
|
|
||
| #### Builtin extensions |
There was a problem hiding this comment.
This has been part of the other RFC, but for me it's still not so clear that we can have "builtin extensions". How do we publish these to PyPI from the main repository? How do we get them to show up on https://extensions.saltproject.io/?
There was a problem hiding this comment.
These "builtin extensions" I introduced it as an optional way for us to be able to provide extensions that do not have yet a proper Salt Extension package.
This could happen for example in these cases:
- Not officially published as Salt Extensions yet (sources would come from "salt-extensions-holding" repository) that we want to include but do not want to maintain upstream.
- Salt Extensions published but not packaged yet in OBS. (in this case we should probably go an package it p)
might not be necessary but it is an option we have.
Builtin extenions would reduce, if necessary, the number of Salt Extensions packages we want to deal with.
It would be up to us to decide whether we want to have any builtin extensions or we just go with all needed extensions as separeted packages if they are already migrated to Salt Extensions.
There was a problem hiding this comment.
In the other RFC, the list of built-in extension includes zypperpkg and transactional_update. These are extensions we want to maintain, i.e. we will need to publish them on PyPI. What's the workflow for maintaining these built-in extensions? Do we have the same rules wrt. pull requests and what they have to contain? How do we publish them to PyPI? What's the plan here?
| For the Salt Extensions that are packaged separately from the main Salt package, we will create a separated GitHub repository where we will maintain these extensions. | ||
|
|
||
| This "openSUSE/salt-extensions" repository will contain: | ||
| - a common salt-extension spec file that will generate all RPM packages |
There was a problem hiding this comment.
So if I get one single github and OBS projects, which will produce one package per each extension. Will this produce/require a big spec file?
There was a problem hiding this comment.
The spec file will be bigger than an if each package is isolated, but it won't be that big actually:
You can see how the spec file looks for 2 packages extensions (mysql and prometheus) here:
https://build.opensuse.org/projects/home:PSuarezHernandez:tests/packages/salt-extensions/files/salt-extensions.spec?expand=1
There was a problem hiding this comment.
ok, looks fine. But it has a high risk of growing a lot, depending on the number of extensions being packaged.
There was a problem hiding this comment.
To me this looks harder to maintain over time than separate packages. With py2pack, it's pretty easy to create new Python packages. Maybe we can it to create new rpms for Salt extensions?
|
|
||
| 1. Stick to our current workflow based on "salt-packaging" -> The workflow doesn't currently fit with Salt Extensions and we don't want to have different workflows between Salt and Salt Extensions. | ||
| 1. One dedicated GitHub repository and OBS package per each Salt Extension -> It won't save resources and will cause more submissions. | ||
| 2. The usage of "git submodules" as an alternative to adding the Salt Extensions sources manually would make it tricky to generate patches manually and also to integrate with "obs_scm". |
There was a problem hiding this comment.
Can you expand on those points a bit? In particular, what makes it tricky to generate patches when sources live in separate repositories opposed to separated sub-directories in a single repository? And why is that harder to integrate separate repositories?
I'm not saying we should go that way, I just want to understand the trade-offs.
There was a problem hiding this comment.
Let say we need to prepare a patch in OBS/IBS to fix some code in one extension or even in many of them in a single shot.
Assuming that we use "git submodules" in our Salt Extension repository and we have a sub-directory per extensions which is a "git submodule":
Then if you run git format-patch command on your Git repository root, then you won't get any diff for any of the "git submodules". You have to run git format-patch inside each "git submodule" directory to get a patch file which is actually not relative to your main Git repo but to the submodule repository, so we won't be able to apply that generated patch directly in our spec file but we would need to manually adjust it.
This is why I say that "git submodules" are not really straight forward when it come to generate patches.
When it comes to the obs_scm service (as we want to have a unified workflow for Salt and its extensions), I have not really tested how it behaves with "git submodules".
Hth!
There was a problem hiding this comment.
Thank you, I understand what you mean now. I agree that using sub-modules in that way is not a good approach. I don't really get the point about patches though, I thought we don't create them anymore with this RFC?
JFI, in the new "SUSE Packaging Git Workflow" git sub-modules are used in "project repositories". Each package that's inside the project is it's own git repository (somewhere else) and they exist inside the project as sub-modules.
There was a problem hiding this comment.
For the general mainteance, we don't use patches anymore but there are some cases where patches are still needed, like the embargoed bugs, where the fixes are manually pushed to IBS, using a patch file, and not via GitHub repository until the bug is public.
| This "openSUSE/salt-extensions" repository will contain: | ||
| - a common salt-extension spec file that will generate all RPM packages | ||
| - The sources for each Salt Extension we package | ||
| - A changelog file |
There was a problem hiding this comment.
Here we'll need to be careful with merge conflicts. Maybe it won't be a big issue, we probably won't change the extensions that often. On the other hand, we probably update them in batches which could lead to changelog merge conflicts.
There was a problem hiding this comment.
Actually, I just changed this as we would have multiple changelog files, one per maintained workflow.
Of course, merge conflicts could happen for PRs that are introducing different changelog entries at the same, then we would need to rebase the PR before merging it.
There was a problem hiding this comment.
Thank you for the updates. For me there are two areas that are not very clear, it would be great if we could clarify them:
- How do we maintain "our" salt extensions?
- The current state of the RFC includes two places that will likely cause merge conflicts on every pull request.
|
|
||
| For the regular Salt maintenance, this means it won't be needed anymore to manually produce patch files to add them to the spec file, as the tarball now contains the updated sources (with the exception of EMBARGOED bugs, where patches are still needed as we cannot push any fix to public GitHub repositories). | ||
|
|
||
| To avoid losing the useful labeling of "PATCH-FIX_UPSTREAM" and "PATCH-FIX_OPENSUSE" (with a direct link to the origin PR on the spec file for each new patch we introduced into our Salt package), we will keep adding this information to the spec file on every new PR but this time without adding the patch itself, only the comment. |
There was a problem hiding this comment.
Won't this cause a lot of merge conflicts?
|
|
||
| When creating a PR to `openSUSE/salt` the user must also include the corresponding changes to the changes file, that can be generated as usual with `osc vc`. | ||
|
|
||
| Similarly to the main Uyuni repository, we should add a GitHub action to warn the user in case no changelog entry is added in the PR. |
There was a problem hiding this comment.
I can't find it, can you help me out with the line number or a link? With the list of "backported PRs" in the spec file, we might hit this issue in two places.
| salt.spec | ||
| ``` | ||
|
|
||
| Since services are disabled here, to allow submissions to openSUSE and SLE, this OBS package will be automatically synced with `openSUSE/release/xxxx` by a Jenkins job. |
There was a problem hiding this comment.
Something is unclear about the workflow to me. You write that we use systemsmanagement:saltstack:github/salt as the devel project. Does that mean the changes go "Github" -> systemsmanagement:saltstack:github/salt -> systemsmanagement:saltstack/salt -> openSUSE:Factory? What exactly is this Jenkins job doing?
There was a problem hiding this comment.
Does that mean the changes go "Github" ->
systemsmanagement:saltstack:github/salt->systemsmanagement:saltstack/salt->openSUSE:Factory?
Yes, that's correct. Essentially:
systemsmanagement:saltstack:github/salt-> services must be enabled here in order to automatically gets the changes from the corresponding GitHub branch, therefore we cannot use this source to submit toFactory(services MUST be disabled when submitting). (poc example: https://build.opensuse.org/package/show/home:PSuarezHernandez:tests:github/salt)systemsmanagement:saltstack/salt-> this is expected to contain same codebase as the above package but in a way that is "ready-to-be-submitted". This means, with the services disabled, and containing the actual spec file, changelog and obsinfo/obscpio files. (poc example: https://build.opensuse.org/package/show/home:PSuarezHernandez:tests/salt)
What exactly is this Jenkins job doing?
The jenkins jobs takes care of automatically pushing any new change that is added to systemsmanagement:saltstack:github/salt via OBS services to the package "ready-to-be-submitted" package at systemsmanagement:saltstack/salt. It disables the services by adjusting the _service, then it manually runs the services to get the spec file, changelog and obsinfo/obscpio files. So after Jenkins runs, the package at systemsmanagement:saltstack/salt contains the same codebase than the :github package and it is ready to be submitted, i.a. to openSUSE:Factory or Maintenance.
I'll try to clarify this better on the RFC text.
|
|
||
| ### Salt Extensions | ||
|
|
||
| #### Builtin extensions |
There was a problem hiding this comment.
In the other RFC, the list of built-in extension includes zypperpkg and transactional_update. These are extensions we want to maintain, i.e. we will need to publish them on PyPI. What's the workflow for maintaining these built-in extensions? Do we have the same rules wrt. pull requests and what they have to contain? How do we publish them to PyPI? What's the plan here?
|
openSUSE:Factory started the move to the Git workflow for the devel projects: https://en.opensuse.org/openSUSE:OBS_to_Git Before me merge this PR, we should make sure that our workflow is compatible with the new Git workflow. From what I can understand, we probably want to have something in src.opensuse.org |
| 1. Stick to our current workflow based on "salt-packaging" -> The workflow doesn't currently fit with Salt Extensions and we don't want to have different workflows between Salt and Salt Extensions. | ||
| 2. One dedicated GitHub repository and OBS package per each Salt Extension -> It won't save resources and will cause more submissions. | ||
| 3. The usage of "git submodules" as an alternative to adding the Salt Extensions sources manually would make it tricky to generate patches manually and also to integrate with "obs_scm". | ||
| 4. Use OBS `scmsync` integration -> while this allows to integrate even the `_service` file into the GitHub repo, it does not work well with SCM/CI workflow, therefore we cannot test the build of the package for an open PR because the branched package is not reflecting the actual changes on the PR. |
There was a problem hiding this comment.
The current approach uses a combination of Jenkins and the OBS<>Github integration, but we still rely on Jenkins as you point out in #Drawbacks. Do we win much by this setup? I wonder if a setup that does not use OBS' SCM/CI Github integration, one that just relies on Jenkins for triggering new builds, would be simpler (as in fewer moving parts). Long term, scmsync might be the way to go.
Read the RFC here