From a8616cac98d6b3d29e8842e867b676c622e1724f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pablo=20Su=C3=A1rez=20Hern=C3=A1ndez?= Date: Thu, 14 Nov 2024 16:38:12 +0000 Subject: [PATCH 01/11] Initial text for new maintaining workflow for Salt --- .../0000-salt-new-maintaining-workflow.md | 218 ++++++++++++++++++ 1 file changed, 218 insertions(+) create mode 100644 accepted/0000-salt-new-maintaining-workflow.md diff --git a/accepted/0000-salt-new-maintaining-workflow.md b/accepted/0000-salt-new-maintaining-workflow.md new file mode 100644 index 00000000..aa9af1e1 --- /dev/null +++ b/accepted/0000-salt-new-maintaining-workflow.md @@ -0,0 +1,218 @@ +- Feature Name: An improved workflow for maintaining Salt +- Start Date: 2024-11-14 + +# Summary +[summary]: #summary + +This RFC proposed an improved workflow for maintaining the Salt package for openSUSE/SUSE distributions, and therefore for Uyuni and SUSE Manager. + +# Motivation +[motivation]: #motivation + +Our current workflow for maintaining Salt requires manual user intervention after the changes are merged into our `openSUSE/salt` codebase, in order to make this changes available in OBS. Moreover, the Salt spec file and patches are tracked in a separated GitHub repository `openSUSE/salt-packaging`, that is also used to generate the changelog entries for the final RPM. + +All these steps needs to be performed manually, with the help of some tooling, to eventually create a manual Submit Request to our Salt package in OBS. + +With Salt Extensions appearing now in the upcoming Salt 3008 release, we want to introduce a new workflow that suits better and solves some the limitations we currently have. + +The purpose of this RFC is: +- Define an new workflow for Salt that reduces user intervention to zero after a given PR is merged in `openSUSE/salt` repository until getting the package ready to consume at OBS. +- Make the Salt maintaining workflow aligned with openSUSE practices. +- Provide a workflow that can also work the same way with packaged Salt Extensions. +- Deprecate the usage of `salt-packaging` repository. + +# Detailed design +[design]: #detailed-design + +In this new workflow the `openSUSE/salt` GitHub repository will become the unique source of thrust, and it will contain: + +- Salt codebase +- Packaging artifacts: spec file, changelog and extra sources +- OBS workflow file + +Taking inspiration from the [OBS/SCM integration guide](https://openbuildservice.org/help/manuals/obs-user-guide/cha-obs-scm-ci-workflow-integration), the new workflow will use OBS workflows and GitHub Webhooks to automate pulling the changes from GitHub to OBS. + +In addition to this, as new Jenkins job will take care of making the OBS package ready to be submitted to openSUSE or SLE. + +This is how the proposed OBS structure would look like: + +- `systemsmanagement:saltstack/salt`: + * no services enabled - package ready to be submitted to openSUSE or SLE. + * branched from `systemsmanagement:saltstack:github/salt` +- `systemsmanagement:saltstack:github/salt`: + * services enabled + * package building based on `openSUSE/release/xxxx` GitHub branch. +- `systemsmanagement:saltstack:github:CI:...:PR-XXXX/salt`: + * services enabled + * package building according to PR branch. + * branched and removed automatically from `systemsmanagement:saltstack:github/salt` by OBS workflow. + +### Packaging artifacts + +All current extra "Sources" files in our RPM package, together with spec file and changelog file will go now to a `pkg/suse/` directory in `openSUSE/salt`: + +``` +pkg/suse/README.SUSE +pkg/suse/html.tar.bz2 +pkg/suse/salt-tmpfiles.d +pkg/suse/transactional_update.conf +pkg/suse/update-documentation.sh +pkg/suse/salt.spec +pkg/suse/salt.changes +``` + +This is the place now where all those files will be maintained. + +#### Salt RPM changelog + +As mentioned this is now at `pkg/suse/salt.changes` in `openSUSE/salt` GitHub repo. + +When creating a PR to `openSUSE/salt` the user must also include the corresponding changes to the spec file, that can be generated as usual with `osc vc`. + +Similarly to the main Uyuni repository, we should add a GitHub action to warn the user in case no changelog entry is added in the PR. + +### OBS project structure + +#### `systemsmanagement:saltstack:github/salt` + +This OBS package will only contain `_multibuild` file and a `_service` file that should look like: + +``` + + + https://github.com/openSUSE/salt.git + git + @PARENT_TAG@ + v(.*) + openSUSE/release/xxxx + pkg/suse/salt.* + + + + + *.tar + gz + + +``` + +The rest of the files will be automatically pulled by the service, as they are enabled here. This package will be automatically refreshed by OBS at any new commit at `openSUSE/release/xxxx` branch. + +#### `systemsmanagement:saltstack/salt` + +This OBS package is a branch from `systemsmanagement:saltstack:github/salt`, where we disable the services and manually run them to get the spec file, changelog and obsinfo/obscpio files, so the package can be submitted to openSUSE or SLE. + +The `_service` file should look like: + +``` + + + https://github.com/openSUSE/salt.git + git + @PARENT_TAG@ + v(.*) + openSUSE/devel/master + pkg/suse/salt.* + + + + + *.tar + gz + + +``` +And it should only contain the following files: + +``` +_multibuild +_service +salt-xxxx.obscpio +salt.changes +salt.obsinfo +salt.spec +``` + +Since services are disabled here, to allow submissions to openSUSE and SLE, this OBS package will be automatically synced with `openSUSE/devel/master` by a Jenkins job. + +### OBS and GitHub Webhook integration + +As described in the [SCM/CI Workflow integration guide](https://openbuildservice.org/help/manuals/obs-user-guide/cha-obs-scm-ci-workflow-integration#sec-obs-obs-scm-ci-workflow-integration-setup-token-authentication-how-to-authenticate-obs-with-scm), a "GitHub Personal Access Token" must be created and a "GitHub Webhook" configure at `openSUSE/salt` repository. + + +#### OBS workflow file + +A `.obs/workflows.yml` will be also added to `openSUSE/salt` to define the OBS workflow as the following: + +``` +main_workflow: + steps: + - branch_package: + source_project: systemsmanagement:saltstack:github + source_package: salt + target_project: systemsmanagement:saltstack:github:CI + filters: + event: pull_request + +rebuild_master: + steps: + - trigger_services: + project: systemsmanagement:saltstack:github + package: salt + filters: + event: push + branches: + only: + - openSUSE/release/xxxx +``` + +This workflow will take care of: + +- Setting up a new subproject at `systemsmanagement:saltstack:github:CI:...:PR-XXXX/salt` for every incoming PR to build the Salt package according to the changes in the PR. +- Triggering the services at `systemsmanagement:saltstack:github/salt` on any new push to `openSUSE/release/xxxx` to build the package accordingly. + +### Making OBS packages ready to submit + +Since the package at `systemsmanagement:saltstack:github/salt` has "services" enabled, and we cannot enable/disable services using OBS workflows, this means this package is not yet ready to be submitted to openSUSE or SLE, as they don't accept enabled services on their submissions. We must disable the services. + +In order to do this we will use an Jenkins job that monitors when a new build is done at `systemsmanagement:saltstack:github/salt` to trigger the following actions at the main `systemsmanagement:saltstack/salt` package: + +``` +# osc checkout systemsmanagement:saltstack/salt +# cd systemsmanagement:saltstack/salt +# osc service manualrun +# osc commit -m "Push new changes from GitHub" +``` + +This way, we ensure `salt.spec` and `salt.changes` and obscpio/obsinfo files gets upgraded according to latest changes. + +### Proof-of-Concept + +I've implemented this structure and automation here: + +- GitHub repository: https://github.com/meaksh/salt/ (`openSUSE/devel/master` branch) +- OBS: + * https://build.opensuse.org/package/show/home:PSuarezHernandez:tests/salt + * https://build.opensuse.org/package/show/home:PSuarezHernandez:tests:github/salt + * https://build.opensuse.org/package/show/home:PSuarezHernandez:tests:github:CI:....:PR-XX/salt + +- Example PR: + * https://github.com/meaksh/salt/pull/10 + * https://build.opensuse.org/package/show/home:PSuarezHernandez:tests:github:CI:meaksh:salt:PR-10/salt + +Feel free to open new PRs against `openSUSE/devel/master` to see this in action. + +# Drawbacks +[drawbacks]: #drawbacks + +TBD + +# Alternatives +[alternatives]: #alternatives + +TDB + +# Unresolved questions +[unresolved]: #unresolved-questions + +TBD From b8bc52b0f5c8edb19d4493036c390db642493625 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pablo=20Su=C3=A1rez=20Hern=C3=A1ndez?= Date: Wed, 27 Nov 2024 16:56:14 +0000 Subject: [PATCH 02/11] Add notes and PoC for Salt Extensions --- .../0000-salt-new-maintaining-workflow.md | 36 +++++++++++++++++-- 1 file changed, 33 insertions(+), 3 deletions(-) diff --git a/accepted/0000-salt-new-maintaining-workflow.md b/accepted/0000-salt-new-maintaining-workflow.md index aa9af1e1..bdd1ad2d 100644 --- a/accepted/0000-salt-new-maintaining-workflow.md +++ b/accepted/0000-salt-new-maintaining-workflow.md @@ -47,6 +47,8 @@ This is how the proposed OBS structure would look like: * package building according to PR branch. * branched and removed automatically from `systemsmanagement:saltstack:github/salt` by OBS workflow. +The same OBS structure will apply to all our OBS targets: `products`, `products:testing` and `products:next`. + ### Packaging artifacts All current extra "Sources" files in our RPM package, together with spec file and changelog file will go now to a `pkg/suse/` directory in `openSUSE/salt`: @@ -171,7 +173,7 @@ This workflow will take care of: - Setting up a new subproject at `systemsmanagement:saltstack:github:CI:...:PR-XXXX/salt` for every incoming PR to build the Salt package according to the changes in the PR. - Triggering the services at `systemsmanagement:saltstack:github/salt` on any new push to `openSUSE/release/xxxx` to build the package accordingly. -### Making OBS packages ready to submit +### Making OBS packages ready to be submitted to Maintenance Since the package at `systemsmanagement:saltstack:github/salt` has "services" enabled, and we cannot enable/disable services using OBS workflows, this means this package is not yet ready to be submitted to openSUSE or SLE, as they don't accept enabled services on their submissions. We must disable the services. @@ -202,15 +204,43 @@ I've implemented this structure and automation here: Feel free to open new PRs against `openSUSE/devel/master` to see this in action. +### Salt Extensions + +#### Builtin extensions +The sources for the builtin Salt Extensions will be located together with the main Salt codebase at the `openSUSE/salt` GitHub repository. No new packages or subpackages will be created for these extensions as they will be part of the main `python3*-salt` package. + +If a fix is needed for any of the builtin extensions, workflow would be the same as for a code fix in the main Salt package. + +#### Packaged Salt Extensions + +For the Salt Extensions that are packaged separately from the main Salt package, we will create a separated GitHub repository where we will maintain these extensions. + +This "openSUSE/salt-extensions" repository will contain: +- a common salt-extension spec file that will generate all RPM packages +- The sources for each Salt Extension we package +- A changelog file +- OBS workflow file + +When it comes to OBS, we will use the same SCM integration and OBS subprojects schema than the proposed for the main Salt codebase. Unique workflow for Salt and Salt Extensions. + +- PoC: + - https://github.com/meaksh/test-repo-1 + - https://build.opensuse.org/package/show/home:PSuarezHernandez:tests/salt-extensions + +NOTE: We are using a single GitHub repo and single OBS package which provides all different salt-extensions RPM packages. This is preferred against having a separated GitHub repositories and OBS package for each Salt Extension, as it will reduce the number of submissions, maintenance incidents and resources needed. + # Drawbacks [drawbacks]: #drawbacks -TBD +- A bit more complex OBS structure than the current one. Including `obs_scm` service. +- Having to still rely on Jenkins to get the packages ready to be released. # Alternatives [alternatives]: #alternatives -TDB +1. Stick to our current workflow based on "salt-packaging" -> The workflow doesn't currently fit with Salt Extensions and we don't want to have different workflows between Salt and Salt Extensions. +1. One dedicated GitHub repository and OBS package per each Salt Extension -> It won't save resources and will cause more submissions. +2. The usage of "git submodules" as an alternative to adding the Salt Extensions sources manually would make it tricky to generate patches manually and also to integrate with "obs_scm". # Unresolved questions [unresolved]: #unresolved-questions From 69aa9fea8f87095b47d771ef012687a4a3850adc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pablo=20Su=C3=A1rez=20Hern=C3=A1ndez?= Date: Wed, 27 Nov 2024 16:58:11 +0000 Subject: [PATCH 03/11] Extra notes about PoC --- accepted/0000-salt-new-maintaining-workflow.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/accepted/0000-salt-new-maintaining-workflow.md b/accepted/0000-salt-new-maintaining-workflow.md index bdd1ad2d..083ae1ff 100644 --- a/accepted/0000-salt-new-maintaining-workflow.md +++ b/accepted/0000-salt-new-maintaining-workflow.md @@ -224,8 +224,12 @@ This "openSUSE/salt-extensions" repository will contain: When it comes to OBS, we will use the same SCM integration and OBS subprojects schema than the proposed for the main Salt codebase. Unique workflow for Salt and Salt Extensions. - PoC: - - https://github.com/meaksh/test-repo-1 - - https://build.opensuse.org/package/show/home:PSuarezHernandez:tests/salt-extensions + * https://github.com/meaksh/test-repo-1 + * https://build.opensuse.org/package/show/home:PSuarezHernandez:tests/salt-extensions + +- Example PR: + * https://github.com/meaksh/test-repo-1/pull/6 + * https://build.opensuse.org/project/show/home:PSuarezHernandez:tests:github:CI:meaksh:test-repo-1:PR-6 NOTE: We are using a single GitHub repo and single OBS package which provides all different salt-extensions RPM packages. This is preferred against having a separated GitHub repositories and OBS package for each Salt Extension, as it will reduce the number of submissions, maintenance incidents and resources needed. From 350a21fe406ab9839db9b874f6c5bc5da650e561 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pablo=20Su=C3=A1rez=20Hern=C3=A1ndez?= Date: Thu, 28 Nov 2024 09:45:01 +0000 Subject: [PATCH 04/11] Fix wrong reference to file --- accepted/0000-salt-new-maintaining-workflow.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/accepted/0000-salt-new-maintaining-workflow.md b/accepted/0000-salt-new-maintaining-workflow.md index 083ae1ff..3bf6d879 100644 --- a/accepted/0000-salt-new-maintaining-workflow.md +++ b/accepted/0000-salt-new-maintaining-workflow.md @@ -69,7 +69,7 @@ This is the place now where all those files will be maintained. As mentioned this is now at `pkg/suse/salt.changes` in `openSUSE/salt` GitHub repo. -When creating a PR to `openSUSE/salt` the user must also include the corresponding changes to the spec file, that can be generated as usual with `osc vc`. +When creating a PR to `openSUSE/salt` the user must also include the corresponding changes to the changes file, that can be generated as usual with `osc vc`. Similarly to the main Uyuni repository, we should add a GitHub action to warn the user in case no changelog entry is added in the PR. From c81cd74ca0de0216e2f5fc535e8cad1c28018da5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pablo=20Su=C3=A1rez=20Hern=C3=A1ndez?= Date: Fri, 29 Nov 2024 12:48:21 +0000 Subject: [PATCH 05/11] Update accepted/0000-salt-new-maintaining-workflow.md Co-authored-by: Marek Czernek --- accepted/0000-salt-new-maintaining-workflow.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/accepted/0000-salt-new-maintaining-workflow.md b/accepted/0000-salt-new-maintaining-workflow.md index 3bf6d879..3c473b3f 100644 --- a/accepted/0000-salt-new-maintaining-workflow.md +++ b/accepted/0000-salt-new-maintaining-workflow.md @@ -24,7 +24,7 @@ The purpose of this RFC is: # Detailed design [design]: #detailed-design -In this new workflow the `openSUSE/salt` GitHub repository will become the unique source of thrust, and it will contain: +In this new workflow the `openSUSE/salt` GitHub repository will become the unique source of trust, and it will contain: - Salt codebase - Packaging artifacts: spec file, changelog and extra sources From c342ef255630ceaf82248d94a54cf4d1eb3ce5ab Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pablo=20Su=C3=A1rez=20Hern=C3=A1ndez?= Date: Fri, 29 Nov 2024 12:48:44 +0000 Subject: [PATCH 06/11] Fix mistake on branch naming --- accepted/0000-salt-new-maintaining-workflow.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/accepted/0000-salt-new-maintaining-workflow.md b/accepted/0000-salt-new-maintaining-workflow.md index 3c473b3f..07e1c042 100644 --- a/accepted/0000-salt-new-maintaining-workflow.md +++ b/accepted/0000-salt-new-maintaining-workflow.md @@ -113,7 +113,7 @@ The `_service` file should look like: git @PARENT_TAG@ v(.*) - openSUSE/devel/master + openSUSE/release/xxxx pkg/suse/salt.* @@ -135,7 +135,7 @@ salt.obsinfo salt.spec ``` -Since services are disabled here, to allow submissions to openSUSE and SLE, this OBS package will be automatically synced with `openSUSE/devel/master` by a Jenkins job. +Since services are disabled here, to allow submissions to openSUSE and SLE, this OBS package will be automatically synced with `openSUSE/release/xxxx` by a Jenkins job. ### OBS and GitHub Webhook integration From 1abc6b04b103a21bd56700119a0dd0b0691b27f8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pablo=20Su=C3=A1rez=20Hern=C3=A1ndez?= Date: Fri, 29 Nov 2024 16:42:12 +0000 Subject: [PATCH 07/11] Add minor rephrase and extra notes about OBS structure --- accepted/0000-salt-new-maintaining-workflow.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/accepted/0000-salt-new-maintaining-workflow.md b/accepted/0000-salt-new-maintaining-workflow.md index 07e1c042..3d5dd258 100644 --- a/accepted/0000-salt-new-maintaining-workflow.md +++ b/accepted/0000-salt-new-maintaining-workflow.md @@ -38,7 +38,7 @@ This is how the proposed OBS structure would look like: - `systemsmanagement:saltstack/salt`: * no services enabled - package ready to be submitted to openSUSE or SLE. - * branched from `systemsmanagement:saltstack:github/salt` + * linked to `systemsmanagement:saltstack:github/salt` - `systemsmanagement:saltstack:github/salt`: * services enabled * package building based on `openSUSE/release/xxxx` GitHub branch. @@ -47,7 +47,11 @@ This is how the proposed OBS structure would look like: * package building according to PR branch. * branched and removed automatically from `systemsmanagement:saltstack:github/salt` by OBS workflow. -The same OBS structure will apply to all our OBS targets: `products`, `products:testing` and `products:next`. +The same OBS structure will apply these other OBS targets, allowing us to deal with different Salt versions if necessary ensuring the packages are also ready to be consumed, without enabled services that could run unexpectely on targets that are linked to them (like i.a. `systemsmanagement:Uyuni:Master`): +- `systemsmanagement:saltstack:products:testing` +- `systemsmanagement:saltstack:products:next` + +For `systemsmanagement:saltstack:products` OBS target, it is not really necessary to follow the above structure, as this target gets updated once we run our "Salt Promote pipeline" (which does copypac whatever is in `products:testing` to `products`). ### Packaging artifacts @@ -102,7 +106,7 @@ The rest of the files will be automatically pulled by the service, as they are e #### `systemsmanagement:saltstack/salt` -This OBS package is a branch from `systemsmanagement:saltstack:github/salt`, where we disable the services and manually run them to get the spec file, changelog and obsinfo/obscpio files, so the package can be submitted to openSUSE or SLE. +This is our ready-to-consume OBS package. It is linked to `systemsmanagement:saltstack:github/salt`, but here we disable the services and manually run them to get the spec file, changelog and obsinfo/obscpio files, so the package can be submitted to openSUSE or SLE. The `_service` file should look like: From 85f3e08a81d2e6d824258962d27c13b2becf1d65 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pablo=20Su=C3=A1rez=20Hern=C3=A1ndez?= Date: Tue, 10 Dec 2024 13:37:45 +0000 Subject: [PATCH 08/11] Add workflow for different changelogs and SCMSYNC integration --- .../0000-salt-new-maintaining-workflow.md | 83 ++++++++++++++----- 1 file changed, 64 insertions(+), 19 deletions(-) diff --git a/accepted/0000-salt-new-maintaining-workflow.md b/accepted/0000-salt-new-maintaining-workflow.md index 3d5dd258..d7831fd4 100644 --- a/accepted/0000-salt-new-maintaining-workflow.md +++ b/accepted/0000-salt-new-maintaining-workflow.md @@ -13,7 +13,7 @@ Our current workflow for maintaining Salt requires manual user intervention afte All these steps needs to be performed manually, with the help of some tooling, to eventually create a manual Submit Request to our Salt package in OBS. -With Salt Extensions appearing now in the upcoming Salt 3008 release, we want to introduce a new workflow that suits better and solves some the limitations we currently have. +With Salt Extensions appearing now in the upcoming Salt 3008 release, we want to introduce a new workflow that suits better and solves some the limitations we currently have. The purpose of this RFC is: - Define an new workflow for Salt that reduces user intervention to zero after a given PR is merged in `openSUSE/salt` repository until getting the package ready to consume at OBS. @@ -27,7 +27,7 @@ The purpose of this RFC is: In this new workflow the `openSUSE/salt` GitHub repository will become the unique source of trust, and it will contain: - Salt codebase -- Packaging artifacts: spec file, changelog and extra sources +- Packaging artifacts: spec file, changelogs, \_multibuild file and extra sources - OBS workflow file Taking inspiration from the [OBS/SCM integration guide](https://openbuildservice.org/help/manuals/obs-user-guide/cha-obs-scm-ci-workflow-integration), the new workflow will use OBS workflows and GitHub Webhooks to automate pulling the changes from GitHub to OBS. @@ -55,7 +55,7 @@ For `systemsmanagement:saltstack:products` OBS target, it is not really necessar ### Packaging artifacts -All current extra "Sources" files in our RPM package, together with spec file and changelog file will go now to a `pkg/suse/` directory in `openSUSE/salt`: +All current extra "Sources" files in our RPM package will go now to a `pkg/suse/` directory in `openSUSE/salt`, together with the spec file, the different maintained changelogs files: ``` pkg/suse/README.SUSE @@ -63,25 +63,67 @@ pkg/suse/html.tar.bz2 pkg/suse/salt-tmpfiles.d pkg/suse/transactional_update.conf pkg/suse/update-documentation.sh +pkg/suse/mkchlog.sh +pkg/suse/_multibuild pkg/suse/salt.spec -pkg/suse/salt.changes +pkg/suse/changelogs/factory/salt.changes +pkg/suse/changelogs/sles15sp2/salt.changes +pkg/suse/changelogs/sles15sp3/salt.changes +pkg/suse/changelogs/sles15sp4/salt.changes +pkg/suse/changelogs/sles15sp5/salt.changes ``` This is the place now where all those files will be maintained. -#### Salt RPM changelog +#### Salt RPM changelogs -As mentioned this is now at `pkg/suse/salt.changes` in `openSUSE/salt` GitHub repo. +As mentioned, the changelog files are now maintained in the `openSUSE/salt` GitHub repo, under `pkg/suse/changelogs/` directory. -When creating a PR to `openSUSE/salt` the user must also include the corresponding changes to the changes file, that can be generated as usual with `osc vc`. +Our packaging artifacts will contain a `mkchlog.sh`, which is a helper script to generate a changelog entry to all maintained changelog in one shot. Something like this: + +```bash +echo "Generating changelog entry for Salt package" +if ! osc vc _temp.changes; +then + exit 1; +fi + +echo "Update changelog files" +echo >> _temp.changes +echo "$(cat _temp.changes salt.changes)" > salt.changes +git add salt.changes + +for i in $(ls changelogs/*/salt.changes); do + echo "$(cat _temp.changes $i)" > $i + git add $i +done + +rm _temp.changes +``` + +When creating a PR to `openSUSE/salt` the user must also include the corresponding changelog entry for all maintained changelog files. Similarly to the main Uyuni repository, we should add a GitHub action to warn the user in case no changelog entry is added in the PR. +NOTE: I think it is better to decouple commit messages (focus on developers) from changelog entries (focus on users/customers), so I prefer to not use commit messages from "openSUSE/salt" to autogenerate the changelog entries but rather to manually write a meaningful changelog message to be included in your PR as part of your changes. Similarly to what we do in other Uyuni repositories. + ### OBS project structure #### `systemsmanagement:saltstack:github/salt` -This OBS package will only contain `_multibuild` file and a `_service` file that should look like: +This OBS package will be configured as "SCM managed", via Meta configuration, as the following: + +``` + + + <description/> + <scmsync>https://github.com/openSUSE/salt?subdir=pkg/suse/#openSUSE/release/xxxx</scmsync> +</package> +``` + +The `_service` file together with the rest of packaging artifacts at `pkg/suse/` directory will be automatically pulled by `scmsync`. + +After this, the rest of the files will be automatically pulled by the services, as they are enabled inside the `_service` file that should look like: ``` <services> @@ -91,7 +133,7 @@ This OBS package will only contain `_multibuild` file and a `_service` file that <param name="versionformat">@PARENT_TAG@</param> <param name="versionrewrite-pattern">v(.*)</param> <param name="revision">openSUSE/release/xxxx</param> - <param name="extract">pkg/suse/salt.*</param> + <param name="extract">pkg/suse/changelogs/factory/salt.changes</param> </service> <service name="set_version" /> <service name="tar" mode="buildtime"/> @@ -102,13 +144,13 @@ This OBS package will only contain `_multibuild` file and a `_service` file that </services> ``` -The rest of the files will be automatically pulled by the service, as they are enabled here. This package will be automatically refreshed by OBS at any new commit at `openSUSE/release/xxxx` branch. +NOTE: This package will be automatically refreshed by OBS at any new commit at `openSUSE/release/xxxx` branch. #### `systemsmanagement:saltstack/salt` -This is our ready-to-consume OBS package. It is linked to `systemsmanagement:saltstack:github/salt`, but here we disable the services and manually run them to get the spec file, changelog and obsinfo/obscpio files, so the package can be submitted to openSUSE or SLE. +This is our ready-to-consume OBS package. We set the devel package (osc setdevelpackage) for this OBS package to `systemsmanagement:saltstack:github/salt`, but here we disable the services and manually run them to get the spec file, changelog and obsinfo/obscpio files, so the package can be submitted to openSUSE or SLE. -The `_service` file should look like: +The `_service` file here should look like: ``` <services> @@ -118,7 +160,9 @@ The `_service` file should look like: <param name="versionformat">@PARENT_TAG@</param> <param name="versionrewrite-pattern">v(.*)</param> <param name="revision">openSUSE/release/xxxx</param> - <param name="extract">pkg/suse/salt.*</param> + <param name="extract">pkg/suse/salt.spec</param> + <param name="extract">pkg/suse/_multibuild</param> + <param name="extract">pkg/suse/changelogs/factory/salt.changes</param> </service> <service name="set_version" mode="manual" /> <service name="tar" mode="buildtime"/> @@ -196,11 +240,11 @@ This way, we ensure `salt.spec` and `salt.changes` and obscpio/obsinfo files get I've implemented this structure and automation here: -- GitHub repository: https://github.com/meaksh/salt/ (`openSUSE/devel/master` branch) +- GitHub repository: https://github.com/meaksh/salt/ (`openSUSE/devel/master-version-2` branch) - OBS: - * https://build.opensuse.org/package/show/home:PSuarezHernandez:tests/salt - * https://build.opensuse.org/package/show/home:PSuarezHernandez:tests:github/salt - * https://build.opensuse.org/package/show/home:PSuarezHernandez:tests:github:CI:....:PR-XX/salt + * https://build.opensuse.org/package/show/home:PSuarezHernandez:tests/fake-package + * https://build.opensuse.org/package/show/home:PSuarezHernandez:tests:github/fake-package + * https://build.opensuse.org/package/show/home:PSuarezHernandez:tests:github:CI:....:PR-XX/fake-package - Example PR: * https://github.com/meaksh/salt/pull/10 @@ -247,8 +291,9 @@ NOTE: We are using a single GitHub repo and single OBS package which provides al [alternatives]: #alternatives 1. Stick to our current workflow based on "salt-packaging" -> The workflow doesn't currently fit with Salt Extensions and we don't want to have different workflows between Salt and Salt Extensions. -1. One dedicated GitHub repository and OBS package per each Salt Extension -> It won't save resources and will cause more submissions. -2. The usage of "git submodules" as an alternative to adding the Salt Extensions sources manually would make it tricky to generate patches manually and also to integrate with "obs_scm". +2. One dedicated GitHub repository and OBS package per each Salt Extension -> It won't save resources and will cause more submissions. +3. The usage of "git submodules" as an alternative to adding the Salt Extensions sources manually would make it tricky to generate patches manually and also to integrate with "obs_scm". +4. Use OBS `scmsync` integration -> while this allows to integrate even the `_service` file into the GitHub repo, it does not work well with SCM/CI workflow, therefore we cannot test the build of the package for an open PR because the branched package is not reflecting the actual changes on the PR. # Unresolved questions [unresolved]: #unresolved-questions From a8448c9f5c3fd563092c761db1a3df130ff8efbc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pablo=20Su=C3=A1rez=20Hern=C3=A1ndez?= <psuarezhernandez@suse.com> Date: Tue, 10 Dec 2024 15:09:51 +0000 Subject: [PATCH 09/11] Adjust changes around SCMSYNC integration --- .../0000-salt-new-maintaining-workflow.md | 35 +++++++------------ 1 file changed, 13 insertions(+), 22 deletions(-) diff --git a/accepted/0000-salt-new-maintaining-workflow.md b/accepted/0000-salt-new-maintaining-workflow.md index d7831fd4..40db213b 100644 --- a/accepted/0000-salt-new-maintaining-workflow.md +++ b/accepted/0000-salt-new-maintaining-workflow.md @@ -20,7 +20,7 @@ The purpose of this RFC is: - Make the Salt maintaining workflow aligned with openSUSE practices. - Provide a workflow that can also work the same way with packaged Salt Extensions. - Deprecate the usage of `salt-packaging` repository. - + # Detailed design [design]: #detailed-design @@ -90,8 +90,6 @@ fi echo "Update changelog files" echo >> _temp.changes -echo "$(cat _temp.changes salt.changes)" > salt.changes -git add salt.changes for i in $(ls changelogs/*/salt.changes); do echo "$(cat _temp.changes $i)" > $i @@ -111,19 +109,7 @@ NOTE: I think it is better to decouple commit messages (focus on developers) fro #### `systemsmanagement:saltstack:github/salt` -This OBS package will be configured as "SCM managed", via Meta configuration, as the following: - -``` -<package name="salt" project="systemsmanagement:saltstack:github"> - <title/> - <description/> - <scmsync>https://github.com/openSUSE/salt?subdir=pkg/suse/#openSUSE/release/xxxx</scmsync> -</package> -``` - -The `_service` file together with the rest of packaging artifacts at `pkg/suse/` directory will be automatically pulled by `scmsync`. - -After this, the rest of the files will be automatically pulled by the services, as they are enabled inside the `_service` file that should look like: +This OBS package will only contain only a `_service` file that should look like: ``` <services> @@ -133,6 +119,8 @@ After this, the rest of the files will be automatically pulled by the services, <param name="versionformat">@PARENT_TAG@</param> <param name="versionrewrite-pattern">v(.*)</param> <param name="revision">openSUSE/release/xxxx</param> + <param name="extract">pkg/suse/_multibuild</param> + <param name="extract">pkg/suse/salt.spec</param> <param name="extract">pkg/suse/changelogs/factory/salt.changes</param> </service> <service name="set_version" /> @@ -144,6 +132,8 @@ After this, the rest of the files will be automatically pulled by the services, </services> ``` +The rest of the files will be automatically pulled by the service, as they are enabled here. This package will be automatically refreshed by OBS at any new commit at `openSUSE/release/xxxx` branch. + NOTE: This package will be automatically refreshed by OBS at any new commit at `openSUSE/release/xxxx` branch. #### `systemsmanagement:saltstack/salt` @@ -160,8 +150,8 @@ The `_service` file here should look like: <param name="versionformat">@PARENT_TAG@</param> <param name="versionrewrite-pattern">v(.*)</param> <param name="revision">openSUSE/release/xxxx</param> - <param name="extract">pkg/suse/salt.spec</param> <param name="extract">pkg/suse/_multibuild</param> + <param name="extract">pkg/suse/salt.spec</param> <param name="extract">pkg/suse/changelogs/factory/salt.changes</param> </service> <service name="set_version" mode="manual" /> @@ -171,7 +161,8 @@ The `_service` file here should look like: <param name="compression">gz</param> </service> </services> -``` +``` + And it should only contain the following files: ``` @@ -240,11 +231,11 @@ This way, we ensure `salt.spec` and `salt.changes` and obscpio/obsinfo files get I've implemented this structure and automation here: -- GitHub repository: https://github.com/meaksh/salt/ (`openSUSE/devel/master-version-2` branch) +- GitHub repository: https://github.com/meaksh/salt/ (`openSUSE/devel/master` branch) - OBS: - * https://build.opensuse.org/package/show/home:PSuarezHernandez:tests/fake-package - * https://build.opensuse.org/package/show/home:PSuarezHernandez:tests:github/fake-package - * https://build.opensuse.org/package/show/home:PSuarezHernandez:tests:github:CI:....:PR-XX/fake-package + * https://build.opensuse.org/package/show/home:PSuarezHernandez:tests/salt + * https://build.opensuse.org/package/show/home:PSuarezHernandez:tests:github/salt + * https://build.opensuse.org/package/show/home:PSuarezHernandez:tests:github:CI:....:PR-XX/salt - Example PR: * https://github.com/meaksh/salt/pull/10 From ae926f906b5b8a768244b0cf2c4766ffda3cb840 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pablo=20Su=C3=A1rez=20Hern=C3=A1ndez?= <psuarezhernandez@suse.com> Date: Tue, 10 Dec 2024 16:13:09 +0000 Subject: [PATCH 10/11] Fix note about changelogs for salt-extensions --- accepted/0000-salt-new-maintaining-workflow.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/accepted/0000-salt-new-maintaining-workflow.md b/accepted/0000-salt-new-maintaining-workflow.md index 40db213b..19cf41dd 100644 --- a/accepted/0000-salt-new-maintaining-workflow.md +++ b/accepted/0000-salt-new-maintaining-workflow.md @@ -257,7 +257,7 @@ For the Salt Extensions that are packaged separately from the main Salt package, This "openSUSE/salt-extensions" repository will contain: - a common salt-extension spec file that will generate all RPM packages - The sources for each Salt Extension we package -- A changelog file +- The changelog files for each maintained codestream - OBS workflow file When it comes to OBS, we will use the same SCM integration and OBS subprojects schema than the proposed for the main Salt codebase. Unique workflow for Salt and Salt Extensions. From a01c6ab702057775ff0c78942288455ee6560b2c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pablo=20Su=C3=A1rez=20Hern=C3=A1ndez?= <psuarezhernandez@suse.com> Date: Wed, 11 Dec 2024 16:42:38 +0000 Subject: [PATCH 11/11] Add section about tracking upstream and downstream patches --- accepted/0000-salt-new-maintaining-workflow.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/accepted/0000-salt-new-maintaining-workflow.md b/accepted/0000-salt-new-maintaining-workflow.md index 19cf41dd..e20f5b9c 100644 --- a/accepted/0000-salt-new-maintaining-workflow.md +++ b/accepted/0000-salt-new-maintaining-workflow.md @@ -75,6 +75,14 @@ pkg/suse/changelogs/sles15sp5/salt.changes This is the place now where all those files will be maintained. +#### Tracking upstream and downstream patches + +This RFC proposes the usage of "obs_scm" service as the mechanism to pull the sources for the Salt package to build. This service will automatically produce a sources tarball according to a configured GitHub branch. + +For the regular Salt maintenance, this means it won't be needed anymore to manually produce patch files to add them to the spec file, as the tarball now contains the updated sources (with the exception of EMBARGOED bugs, where patches are still needed as we cannot push any fix to public GitHub repositories). + +To avoid losing the useful labeling of "PATCH-FIX_UPSTREAM" and "PATCH-FIX_OPENSUSE" (with a direct link to the origin PR on the spec file for each new patch we introduced into our Salt package), we will keep adding this information to the spec file on every new PR but this time without adding the patch itself, only the comment. + #### Salt RPM changelogs As mentioned, the changelog files are now maintained in the `openSUSE/salt` GitHub repo, under `pkg/suse/changelogs/` directory.