Skip to content

Feature/301 create utility scripts to compute the next release build for cbs #310

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 160 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
160 commits
Select commit Hold shift + click to select a range
c8ca5db
Fetch build dependencies based on AD
dennis-behm Jan 30, 2025
82a3809
Missing import of Utilities
dennis-behm Jan 30, 2025
e9eab81
Align fetchDependencies script with Application Descriptor
dennis-behm Jan 30, 2025
8de7794
initial documentation
dennis-behm Jan 30, 2025
f584cfc
initial documentation
dennis-behm Jan 30, 2025
9e98a02
Consolidate fetch script for both dbbBuild.sh and zBuilder.sh
dennis-behm Jan 30, 2025
7b49b4e
Set execute bits for zBuilder.sh
dennis-behm Jan 30, 2025
ca23c00
Update release info and add page break
dennis-behm Feb 3, 2025
98e9de0
Doc updates from code review
dennis-behm Feb 5, 2025
b87255e
Configure Cache directory
dennis-behm Feb 5, 2025
9a1f583
Configure Cache directory
dennis-behm Feb 5, 2025
e923fd2
Configure Cache directory
dennis-behm Feb 5, 2025
8499816
Configure Cache directory
dennis-behm Feb 5, 2025
4a32af5
Add dbbCommunityRepoRoot
dennis-behm Feb 7, 2025
27587cf
Read Application Descriptor via utils
dennis-behm Feb 7, 2025
7e1ac6d
Read Application Descriptor via utils
dennis-behm Feb 7, 2025
88f2df3
Read Application Descriptor via utils
dennis-behm Feb 7, 2025
2881037
Update Packaging process
dennis-behm Feb 24, 2025
d4bd7dd
Update Packaging notes
dennis-behm Feb 24, 2025
4aba150
packageUtils
dennis-behm Feb 24, 2025
43797c7
Add eyecatcher to build frameworks
dennis-behm Feb 7, 2025
4b25d17
why on earth again
dennis-behm Feb 24, 2025
2d9da76
bla
dennis-behm Feb 24, 2025
333c0f7
add buildIdentifier and releaseIdentifier
dennis-behm Feb 28, 2025
2eea9b1
Compute publishing parameters for packaging
dennis-behm Feb 28, 2025
068cc8b
fix duplicate variable usage
dennis-behm Feb 28, 2025
fd398d2
introduce packageBuildIdentifier
dennis-behm Feb 28, 2025
d09cdac
Refactored PackageBuildOutputs to return url
dennis-behm Mar 3, 2025
86a5b61
reference PackagingScript
dennis-behm Mar 3, 2025
31a2439
document external dependencies
dennis-behm Mar 3, 2025
14caa4f
polish cli options
dennis-behm Mar 3, 2025
cb713bb
generalize repo url computation
dennis-behm Mar 3, 2025
725c4f2
switch to updated AD dependency declaration
dennis-behm Mar 3, 2025
cbf88b1
updated reference for testing
dennis-behm Mar 3, 2025
5a8296e
Consider packageUrl config file
dennis-behm Mar 7, 2025
172f7a2
change eyecatcher
dennis-behm Mar 7, 2025
50e9934
new standalone cbs to recompute the package url
dennis-behm Mar 7, 2025
aabd600
set default name for packageVersionConfig
dennis-behm Mar 7, 2025
601dc33
minor fixes
dennis-behm Mar 7, 2025
f8e4b6b
Consolidate package url computation into wazideploy-generate.sh
dennis-behm Mar 10, 2025
1da2b20
merged wazideploy-gen
dennis-behm Mar 10, 2025
e78bf47
update config for testing
dennis-behm Mar 10, 2025
9547523
make WD manifest default
dennis-behm Mar 10, 2025
0a17ed0
move package url computation into ArtifactRepositoryHelpers
dennis-behm Mar 13, 2025
1bf2035
Use computation in ArtifactRepoHelpers
dennis-behm Mar 13, 2025
da200ea
include package url computation in wazideploy-generate
dennis-behm Mar 13, 2025
2740ac7
remove script configuration from ucd integrations
dennis-behm Mar 13, 2025
d439114
remove script configuration from packaging integration
dennis-behm Mar 13, 2025
1c37255
remove script configuration from pipelinebackend config
dennis-behm Mar 13, 2025
52e2ea9
remove script configuration from pipelinebackend config
dennis-behm Mar 13, 2025
d73c755
remove script configuration from pipeline configuration
dennis-behm Mar 13, 2025
9c602a2
consolidate how the url is computed
dennis-behm Mar 13, 2025
15cfcbd
print result to be picked up by cbs
dennis-behm Mar 13, 2025
79f6ca0
return rc on upload
dennis-behm Mar 13, 2025
b8f5edf
return rc on download
dennis-behm Mar 13, 2025
4431915
capture the rc from upload
dennis-behm Mar 13, 2025
53af650
evaluate return code
dennis-behm Mar 13, 2025
ac1c4bc
fix syntax error
dennis-behm Mar 13, 2025
62f4091
switch to suffix pattern
dennis-behm Mar 13, 2025
9f6aab6
Add release update
dennis-behm Mar 13, 2025
ef71fc4
remote testing bits
dennis-behm Mar 13, 2025
4b55eb8
Add release update wdg
dennis-behm Mar 13, 2025
27d008e
fix merge conflict
dennis-behm Mar 13, 2025
226e7f5
remove function
dennis-behm Mar 13, 2025
839fd3a
remove function from config
dennis-behm Mar 13, 2025
eedd6b6
update log outputs
dennis-behm Mar 13, 2025
676226d
updated structure of AD for dependencies
dennis-behm Mar 13, 2025
8d62c0d
align with new dependency attribute in application descriptor
dennis-behm Mar 13, 2025
2d9990a
Doc updates
dennis-behm Mar 17, 2025
89b8f8c
Doc updates
dennis-behm Mar 17, 2025
d55517f
remove slurper
dennis-behm Mar 17, 2025
6b56de8
Use artifact name suffix
dennis-behm Mar 17, 2025
4936889
Change to blanks instead of tab
dennis-behm Mar 17, 2025
70c7551
Cleanup packaging scripts
dennis-behm Mar 17, 2025
d5d0c15
additional documentation
dennis-behm Mar 17, 2025
716c8f3
Align deployment step with download approach
dennis-behm Mar 19, 2025
0e261b1
Pass applicationFolderPath to publish interfaces
dennis-behm Mar 19, 2025
1bf7cd9
add check for tarfile
dennis-behm Mar 19, 2025
9552aa1
update readme
dennis-behm Mar 19, 2025
0b5921e
apply format
dennis-behm Mar 19, 2025
07ff7da
Ensure backward compatibilty passing versionName
dennis-behm Mar 20, 2025
f2e4b89
store package in deployfolder
dennis-behm Mar 20, 2025
8e5e23b
store package in deployfolder fix path
dennis-behm Mar 20, 2025
139a28c
Expect application descriptor on the root level of the application di…
dennis-behm Mar 20, 2025
85ee445
Validate configuration parm
dennis-behm Mar 20, 2025
718a6ea
hide pwds
dennis-behm Mar 20, 2025
7549a5a
remove prefixing
dennis-behm Mar 20, 2025
ad13508
Align with required location of the application descriptor
dennis-behm Mar 20, 2025
5e52119
Document external dependencies in wazi deploy app manifest
dennis-behm Mar 21, 2025
c0cdf1b
update file references
dennis-behm Mar 21, 2025
132e69e
Add script to compute release version
Mar 24, 2025
5bdbd62
Move computeReleaseVersion.sh to CBS main directory
Mar 24, 2025
b899fc1
test computeReleaseVersion.sh
Mar 24, 2025
e0dc379
test computeReleaseVersion.sh
Mar 24, 2025
5c52251
test computeReleaseVersion.sh
Mar 24, 2025
09a7574
Simplify package management
dennis-behm Mar 24, 2025
6f9adce
test computeReleaseVersion.sh
Mar 24, 2025
c4aedd3
Pass Helper location
dennis-behm Mar 24, 2025
70ce2f3
extract the url
dennis-behm Mar 24, 2025
654ae89
extract the url
dennis-behm Mar 24, 2025
72e48e9
test computeReleaseVersion.sh
Mar 24, 2025
c80fb56
test computeReleaseVersion.sh
Mar 24, 2025
cf6c490
test computeReleaseVersion.sh
Mar 24, 2025
945e234
update passing variables
dennis-behm Mar 24, 2025
2df7fcf
Updated options for computeReleaseVersion.sh
Mar 24, 2025
c230415
Updated computeReleaseVersion.sh
Mar 24, 2025
5c45d09
expand fetching baseline package
dennis-behm Mar 24, 2025
6e83355
Debug computeReleaseVersion.sh
Mar 25, 2025
90ded94
manage baseline package during packaging step
dennis-behm Mar 25, 2025
b05ebd8
remove scriptDir
dennis-behm Mar 25, 2025
6d73045
Apply suggestions from code review - primarily documentation
dennis-behm Mar 25, 2025
d8d6eda
Update computeReleaseVersion.sh
Mar 25, 2025
5fd8ddf
Resolve conflict
Mar 25, 2025
daec39d
add description about baseline package
dennis-behm Mar 25, 2025
ae02e69
add description about baseline package
dennis-behm Mar 25, 2025
75311e8
Rename internal method
dennis-behm Mar 25, 2025
1f5f65f
Reviewer feedback to rename variable
dennis-behm Mar 25, 2025
6de3d4c
Reviewer feedback to rename outputs
dennis-behm Mar 25, 2025
b3cfe74
remove unused variable
dennis-behm Mar 25, 2025
942d4a2
remove odd exit path
dennis-behm Mar 25, 2025
824e652
call the utilities instead of sourcing it
dennis-behm Mar 25, 2025
ac0a7cc
pass correct cli to packaging script
dennis-behm Mar 25, 2025
dea582a
fix baseline package reference
dennis-behm Mar 25, 2025
b1bf88e
README updates
dennis-behm Mar 26, 2025
858591b
Apply suggestions from code review
dennis-behm Mar 28, 2025
15504fe
numbering in readme
dennis-behm Mar 28, 2025
1391d58
Switch to UTF-8 output
dennis-behm Mar 28, 2025
deeb9bb
Fix path in ucdPackaging
dennis-behm Mar 31, 2025
e4e41f2
Remove configuration
dennis-behm Apr 1, 2025
61d1d9a
Test computeReleaseVersion
Apr 1, 2025
275a526
Apply suggestions from code review
dennis-behm Apr 1, 2025
51a0da2
Use Archive instead of Package
dennis-behm Apr 1, 2025
9de0df9
Fix renamed object names
dennis-behm Apr 1, 2025
eaca963
sync with latest AD schema
dennis-behm Apr 2, 2025
5da0453
code formatting
dennis-behm Apr 2, 2025
f03c412
Updating messages for more consistency
dennis-behm Apr 2, 2025
3739497
rename to archiveCache
dennis-behm Apr 2, 2025
6134ce3
disable new feature
dennis-behm Apr 2, 2025
be55b7a
rename to just url
dennis-behm Apr 7, 2025
5d73f63
manage missing AD definitions
dennis-behm Apr 7, 2025
28b88c4
fix syntax error
dennis-behm Apr 7, 2025
837e847
fix syntax error
dennis-behm Apr 7, 2025
9cf238f
Merge with Dennis's changes to enable calculation of Artifactory URL
Apr 8, 2025
6ba67ce
Update pipelineBackend.config
Apr 8, 2025
1f98451
Merge with Dennis's changes to enable calculation of Artifactory URL
Apr 8, 2025
288170b
Finalize and add comments to computeReleaseVersion.sh
Apr 11, 2025
c12192d
Add jenkins pipeline definition template that use Wazi Deploy
Apr 11, 2025
71a10e6
Clean up value in pipelineBackend.config
Apr 11, 2025
f53be77
Merge remote-tracking branch 'upstream/main' into feature/301-create-…
Apr 28, 2025
6c553a4
Revert back the file to the original commit
Apr 28, 2025
1848448
Separate the Jenkin template changes for another PR.
Apr 28, 2025
8e3ee4a
Revert file permissions changed back to original
Apr 29, 2025
29e817a
Document computeReleaseVersion.sh script in the readme.md file and mi…
Apr 29, 2025
459c0b0
Update based on the feedback recieved.
Apr 30, 2025
4c04c2b
Update based on the PR comments
May 6, 2025
98381e1
Add instruction/example how to retrieve the version number by pipelin…
May 7, 2025
346886e
Add instruction/example how to retrieve the version number by pipelin…
May 7, 2025
e077634
Add instruction/example how to retrieve the version number by pipelin…
May 7, 2025
76c8b1e
Add instruction/example how to retrieve the version number by pipelin…
May 7, 2025
c22a823
Add instruction/example how to retrieve the version number by pipelin…
May 7, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 83 additions & 5 deletions Templates/Common-Backend-Scripts/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,8 @@ Artifact Name | Description | Script details
---------- | -----| -----------------------------------------------------
[gitClone.sh](gitClone.sh) | Pipeline Shell Script to perform Git Clone to z/OS UNIX System Services | [script details](README.md#41---gitclonesh)
[dbbBuild.sh](dbbBuild.sh) | Pipeline Shell Script to invoke the Dependency Based Build framework [zAppBuild](https://github.com/IBM/dbb-zappbuild) | [script details](#dbbbuildsh-for-zappbuild-frameworkh)
[zBuilder.sh](zBuilder.sh) | Pipeline Shell script to invoke the zBuilder framework [zBuilder](https://www.ibm.com/docs/en/dbb/3.0?topic=building-zos-applications-zbuilder) | [script details](#zbuildersh-for-dbb-zbuilder)
[zBuilder.sh](zBuilder.sh) | Pipeline Shell Script to invoke the zBuilder framework [zBuilder](https://www.ibm.com/docs/en/dbb/3.0?topic=building-zos-applications-zbuilder) | [script details](#zbuildersh-for-dbb-zbuilder)
[computeReleaseVersion.sh](computeReleaseVersion.sh) | Pipeline Shell Script to compute the next release version based on the baseline version information stored in the application's [baselineReference.config](samples/baselineReference.config) file. | [script details](#computeReleaseVersionsh)
[packageBuildOutputs.sh](packageBuildOutputs.sh) | Pipeline Shell Script to create a Package using the [PackageBuildOutputs groovy script](https://github.com/IBM/dbb/tree/main/Pipeline/PackageBuildOutputs) | [script details](#packagebuildoutputssh)
[ucdPackage.sh](ucdPackaging.sh) | Pipeline Shell Script to publish to UCD Code Station binary repository using the [CreateUCDComponentVersion groovy script](https://github.com/IBM/dbb/tree/main/Pipeline/CreateUCDComponentVersion) | [script details](#ucdpackagingsh)
[wazideploy-generate.sh](wazideploy-generate.sh) | Pipeline Shell Script to generate a Deployment Plan to be used with Wazi Deploy | [script details](#wazideploy-generatesh)
Expand Down Expand Up @@ -492,6 +493,84 @@ The [dbbzBuilderUtils](utilities/dbbzBuilderUtils.sh) script is a core utility s

Depending on the Deployment Manager tool you are using, you can choose from either creating a package with the [PackageBuildOutputs](#packagebuildoutputssh) script that can be used with IBM Wazi Deploy, or the [UCD packaging](#ucdpackagingsh) script that creates the UCD shiplist and UCD component version.

### computeReleaseVersion.sh

This script is to compute the version for the next release based on the information of baseline version stored in the [baseineReference.config](samples/baselineReference.config) file. The computation of the release version follows the versioning described in [IBM recommended Git branching model for mainframe developer](https://ibm.github.io/z-devops-acceleration-program/docs/branching/git-branching-model-for-mainframe-dev). It follows the Systematic Versioning as MAJOR.MINOR.PATCH number. The number of the coresponding part is increased based on the type of release.

The computed release version is used as part of the archive name to be uploaded or downloaded from the artifact repository. The release version is an input for [packageBuildOutputs.sh](#packagebuildoutputssh) and [wazideploy-generate.sh](#wazideploy-generatesh).

#### Invocation

The `computeReleaseVersion.sh` script can be invoked as follows:

```
computeReleaseVersion.sh -w MortApp/main/build-1 -a MortgageApplication -b main -r minor
```

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please add a sample how to retrieve the result from the command from a pipeline script?

CLI parameter | Description
---------- | ----------------------------------------------------------------------------------------
-w `<workspace>` | **Workspace directory**, an absolute or relative path that represents unique directory for this pipeline definition, that needs to be consistent through multiple steps.
-a `<application>` | **Application name** to be built.
-b `<branch>` | **Git branch** that is built.
-r `<major/minor/patch>` | **Release Type** to indicate the type of changes to be relased.

#### Output

The section below contains the output that is produced by the `computeReleaseVersion.sh` script.

<details>
<summary>Script Output</summary>

```
+ computeReleaseVersion.sh -w /var/jenkins/workspace/tgageApplication-Pipelines2_main -a MortgageApplication -b main -r major
[Pipeline] echo
computeReleaseVersion.sh: [INFO] Release Version Wrapper. Version=1.10
computeReleaseVersion.sh: [INFO] Reading pipeline configuration file: /var/jenkins/dbb/Templates/Common-Backend-Scripts/pipelineBackend.config
computeReleaseVersion.sh: [INFO] Validating Options
computeReleaseVersion.sh: [INFO] Application Directory: /var/jenkins/workspace/tgageApplication-Pipelines2_main/MortgageApplication
computeReleaseVersion.sh: [INFO] Detected the application respository (MortgageApplication) within the git repository layout structure.
computeReleaseVersion.sh: [INFO] Assuming this as the new application location.
computeReleaseVersion.sh: [INFO] **************************************************************
computeReleaseVersion.sh: [INFO] ** Started Next Release Computation on HOST/USER: z/OS ZT01 05.00 02 8561/
computeReleaseVersion.sh: [INFO] ** Workspace: /var/jenkins/workspace/tgageApplication-Pipelines2_main/MortgageApplication
computeReleaseVersion.sh: [INFO] ** Application: MortgageApplication
computeReleaseVersion.sh: [INFO] ** Branch: main
computeReleaseVersion.sh: [INFO] ** Release Type: major
computeReleaseVersion.sh: [INFO] ** Baselinereference file: /var/jenkins/workspace/tgageApplication-Pipelines2_main/MortgageApplication/application-conf/baselineReference.config
computeReleaseVersion.sh: [INFO] **************************************************************

computeReleaseVersion.sh: [INFO] Baseline reference: refs/tags/rel-1.6.0
computeReleaseVersion.sh: [INFO] Compute the next release version complete. The next release version: rel-2.0.0. rc=0

```

</details>

#### Extract the next release version

To obtain the release version from the `computeReleaseVersion.sh` script's output log, utilize string manipulation commands within your pipeline orchestrator to search for the pattern `version: .*rc=0` pattern and extract the version number form the search result.

<details>
<summary>Example snippet to extract the version number</summary>

```
regexPattern = "version: .*rc=0"
pattern = java.util.regex.Pattern.compile(regexPattern)
//logContent is the output from invoking the computeReleaseVersion.sh script
def pMatcher = pattern.matcher(logContent)
if (pMatcher.find()) {
def version = "${pMatcher.group()}"
if (version){
releaseVersion = version.substring(9,version.length()-6)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hard-coding the length seems to be error-prone.

What about a rel-10.123.876. I did not test it, but I assume this would cut off some information.

Copy link
Member Author

@monvora monvora May 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dennis-behm, in this case if the log message isn't changed, the information will not be lost since the end part is always at the position (string length - 6) because the last part of the log message is always '. RC=0'.

}
} else {
println("[INFO]: Failed to search for ${regexPattern}")
}

```

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@monvora this is missing the closing </details> tag. It looks like the rest of the page is truncated.

https://github.com/IBM/dbb/blob/76c8b1e5e344373e280eabb70f3585c0fd8945fe/Templates/Common-Backend-Scripts/README.md#packaging-stage

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

</details>

### packageBuildOutputs.sh

This script is to execute the `PackageBuildOutputs.groovy` that packages up the build outputs and optionally uploads it to an artifact repository to publish the artifacts created by a DBB build in the pipeline.
Expand All @@ -512,7 +591,8 @@ For release builds (that use the `pipelineType=release`), the archive is uploade
`release/<reference>/<application>-<buildIdentifier>`:

* **release** is defined for release builds.
* **reference** is the release name: for instance, `rel-1.2.3` (provided through the mandatory `-r` argument).
* **reference** is the release name: for instance, `rel-1.2.3` (provided through the mandatory `-r` argument). The release name can be automatically computed using the [computeReleaseVersion.sh](#computereleaseversionsh) script.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we extend the packageBuildOutputs.sh to accept a new flag like -R (requires some analysis), which then calls computeReleaseVersion.sh as a utility program.

This will reduce the necessary communication between the pipeline orchestrator and the mainframe (keeps the pipeline implementation simpler).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dennis-behm I like this idea. The pipeline implementation will definitely simpler. I will made all the changes you suggested above to close this PR. Then, I'll look into extending the packageBUildOutputs.sh script.


The archive's file name is computed using the application's name, the release name (`-r` argument) and a unique build identifier (`-i` argument). This parameter is typically the pipeline build number that is passed by the pipeline orchestrator. If a build identifier is not provided, the current timestamp is used.


Expand Down Expand Up @@ -629,8 +709,6 @@ rc=0

</details>



### ucdPackaging.sh

This script is to execute the `dbb-ucd-packaging.groovy` that invokes the Urban Code Deploy (UCD) buztool utility, to publish the artifacts created by the DBB Build from a pipeline.
Expand Down Expand Up @@ -727,7 +805,7 @@ CLI parameter | Description
-- | - when retrieving the tar file from Artifact repo the below options are mandatory -
-b `<branch>`| Name of the **git branch** turning into a segment of the directory path for the location within the artifact repository.
-p `<build/release>` | **Pipeline Type** to indicate a `build` pipeline (build only with test/debug options) or a `release` pipeline (build for optimized load modules for release candidates).
-R `<releaseIdentifier>` | **Release identifier** to indicate the next planned release name. This is a computed value based on the pipeline templates.
-R `<releaseIdentifier>` | **Release identifier** to indicate the next planned release name. This is a computed value based on the pipeline templates. It can be automatically computed using the [computeReleaseVersion.sh](#computereleaseversionsh) script.
-I `<buildIdentifier>` | **Build identifier** a unique value to identify the tar file. This is a computed value provided by the pipeline templates. Typically the build number of the pipeline run.
-c `<configurationFile>` | Absolute path to the Wazi Deploy **Configuration File** that contains information to connect to the artifact repository. See [IBM Wazi Deploy documentation](https://www.ibm.com/docs/en/developer-for-zos/17.0?topic=files-configuration-file)

Expand Down
Loading