How can external workflows be authored in a way that makes sure best practices are followed?

[peternied]

Release workflows will need to describe what actions should take place when its time to release the product. These will be run only as often as releases occur. How can workflows be authored in a way that makes them durable to changes in OpenSearch-Build repositories changes as well as getting notified about sensible workflow changes to make before the next release?

Acceptance Criteria:

• Determine the smallest size a workflow can be to limit the number of practices that need to be enforced.
  • An example should be created in the documentation
• Define when code should be authored externally vs at the central repository.
  • E.g. a new docker image creation workflow is created, should that be done in the client repo or in the shared repo?
• Define ci-checks that teams should follow

[gaiksaya]

On high level the Jenkins libraries to release the artifacts will be located in opensearch-build repo similar to other jenkins libraries https://github.com/opensearch-project/opensearch-build/tree/main/vars . In this way we can reuse the existing codebase (like for signing).

However in case of JenkinsFiles and its regression test files we have two options: (Need to choose one)

Option 1:

The jenkins files and its regression test files for each component will be situated in the respective component’s github repository.

Pros:

• In this way the component owners have the control over their release workflow
• Branching is supported out of the box, meaning components can incrementally improve their processes as per the version
• No dependency on code owners for code approvals, etc.

Cons:

• The code change across the repositories can cause breaking changes:
  • Jenkins Lib is changed in opensearch-build repo: Since the regressions files for all client’s workflows will be situated in their own repository, the changes won’t be detected until very late when some CI runs the workflow in client’s repo
  • JenkinsFile changes in component repo: The change needs to be conveyed to build repo code owner team in case of any variable, secrets addition, infrastructure change (like new agent requirement), etc.

Option 2:

Everything is located in opensearch-build repo.

Pros

• One place for all jenkinsFiles, libraries and respective tests
• The changes need not be propagated across the repositories

Cons:

• Component team has less control over the release workflow
• Chances of build repo code owner team being the bottle neck for code approvals to release workflows (which is okay since the infrastructure is owned by them so any breaking change can be detected in the review)
• Component team would not be aware of any changes in the central jenkins libraries that affects component’s release workflow too.

[dblock]

Because of branching I would choose option 1. I think the Jenkins library should be split up in smaller parts (e.g. signing), each part versioned, and released as a standalone component, similar to GitHub Actions, to solve the cons.

[gaiksaya]

I am not sure that is doable or not @dblock . Was researching regarding jenkins shared libraries and all the content I found is asking to use the git branch, tag or commit as version. GitHub Actions has each action has separate git repository and that's why its easier to version per action.

Adding @prudhvigodithi to see if he knows of any such way.

[gaiksaya]

Researched a bit on pulling different versions for remote jenkins shared libraries, and found this plugin https://plugins.jenkins.io/workflow-cps-global-lib-http/
Discussed with @prudhvigodithi, we can publish the zips to maven or any other artifact repository (s3 works as well) for different versions and accordingly pull the required version.

Few things to keep in mind:

1. Jenkins Shared Library versions will be independent of opensearch and other product's version. (should we separate this out into new repository?)
2. Version release notes, change log needs to be maintained to let the consumer of this library know what changed and if the next version can cause any breaking changes.
3. Versioning should follow semver

[gaiksaya]

• Determine the smallest size a workflow can be to limit the number of practices that need to be enforced.
  Will be added as a part of documentation issue

• Define when code should be authored externally vs at the central repository.
  See Option 1
  The cons can be resolved by 1. versioning the shared libraries. See issue 2. Adding OSEE team to approve PRs wherever required.

• Define ci-checks that teams should follow
  All jenkins libraries as well as jenkinsFile should have regression test.
  More details on how to add them can be found here https://github.com/opensearch-project/opensearch-build/blob/main/DEVELOPER_GUIDE.md#regression-tests
  More details will be added during documenting everything. See documentation meta issue Documentation meta issue #2493