Revised implementation of reporting of external impacts feature (IBM#239

) * Revised implementation of reporting of external impacts Signed-off-by: Dennis Behm <[email protected]> * Revisions to Function Overview of Report External Impacts Signed-off-by: Lauren Li <[email protected]> * Handle case with back-level DBB server Signed-off-by: Dennis Behm <[email protected]> Signed-off-by: Dennis Behm <[email protected]> Signed-off-by: Lauren Li <[email protected]> Co-authored-by: Lauren Li <[email protected]>
dennis-behm · Aug 24, 2022 · 6c9288d · 6c9288d
1 parent 3785ac8
commit 6c9288d
Show file tree

Hide file tree

Showing 4 changed files with 336 additions and 214 deletions.
diff --git a/REPORT.md b/REPORT.md
@@ -18,9 +18,22 @@ The _Reporting External Impacts_ feature enables the build framework to generate
 
 The reports is meant to be used to start the collaboration process with the other application teams about their adoption process of the shared resource, typically a copybook. A future idea is to use the reports to add files to the build list so that the consuming application can built their impacted files.
 
-### Pre-requisites
+### Functional Overview
+
+Technically, this feature analyzes the files of the calculated and provided build list including the identified changed files of the current pipeline execution. Based on this list, the feature queries the collections of other applications within the DBB WebApp. To fully analyze cross application impacts, other applications must also be part of and be processed through the CI/CD pipeline with DBB.
+
+This feature is available on all build types leveraging a DBB repository client connection, such as `--impactBuild`, `--mergeBuild` or `--fullBuild`. With the latest updates, it requires both DBB toolkit version and DBB Webapp versions to be on at least on 1.1.3.
+
+It can operate in two modes: Simple and Deep.
+
+**Simple** mode allows the identification and report of directly-impacted files. Supported scenarios include the following:
+
+* Sample Scenario 1 - Copybook COPYA in application App-A is referenced by programs in applications App-B and App-C. If copybook COPYA is changed, the reporting of external impacts will document the impacted programs of App-B and App-C.
+* Sample Scenario 2 - A submodule SUBPGMA in application App-A is included as an object deck in multiple other applications through processing a linkcard (static linkage). If SUBPGMA is changed, then the reporting of external impacts will document the programs of the other impacted applications. In this case, the submodule SUBPGMA can even be an impacted file of a copybook change in App-A.
+
+**Deep** mode performs the simple analysis first, and then passes its results back into the analysis process to further find files that are impacted by the already-identified impacted files.
 
-Technically, this feature analyzes the list of changed files for the current application and then queries the collections of other applications within the DBB WebApp. Therefore, to fully analyze cross application impacts, the other applications must also be part of and processed through the CI/CD pipeline with DBB. It requires a DBB WebApp repository connection.
+* Sample Scenario 1 - A changed copybook COPYA in application App-A is referenced by submodules in applications App-B and App-C. The deep reporting will first identify the impacted submodules in App-B and App-C, and then use those results in its extended analysis to identify the impacted linkcards used to create the binaries in App-B and App-C.
 
 ### Configuration
 
@@ -33,24 +46,65 @@ In the below sample, the MortgageApplication was split into two applications App
 
 App-EPSM provides a shared copybook `epsmtcom.cpy`, which is included in programs of App-EPSC. While the overall build strategy does not intend to force an automated rebuild in App-EPSC for the changed shared artifact, however would like the capability to provide a report to the application team, which owns this shared element.
 
-The build console output for App-EPSM will contain the below section (with `--verbose`):
+The build console output for App-EPSM will contain the below section (with `--verbose` option):
 
 ```
-** Analyse and report external impacted files.
-*** Identified external impacted files for changed file App-EPSM/copybooks-public/epsmtcom.cpy
-*** Writing report of external impacts to file /var/jenkins/workspace/App-EPSM-S4/outputs/build.20210722.032455.024/externalImpacts_App-EPSC-master.txt
+...
+** Writing build list file to /u/jenkins/workspace/App-EPSM/build.20220810.122244.022/buildList.txt
+App-EPSM/cobol/epsplist.cbl
+** Perform analysis and reporting of external impacted files for the build list including changed files.
+*** Running external impact analysis with file filter **/* and collection patterns .*-main.*,.*-develop.* with analysis mode simple
+*** Running external impact analysis for files 
+     App-EPSM/copybooks-public/epsmtcom.cpy 
+     App-EPSM/cobol/epsplist.cbl 
+*** Potential external impact found EPSCSMRT (App-EPSC/cobol/epscsmrt.cbl) in collection App-EPSC-main 
+*** Potential external impact found EPSMLIST (App-EPSC/cobol/epsmlist.cbl) in collection App-EPSC-main 
+*** Potential external impact found EPSCMORT (App-EPSC/cobol/epscmort.cbl) in collection App-EPSC-main 
+*** Writing report of external impacts to file /u/jenkins/workspace/App-EPSM/build.20220810.122244.022/externalImpacts_App-EPSC-main.txt
+** Invoking build scripts according to build order: BMS.groovy,Cobol.groovy,LinkEdit.groovy
+...
 ```
 
-For each collection in scope of the analysis with impacted files a report is generated and written (in this case `externalImpacts_App-EPSC-master.txt`) to the `workdir`. A report will contain three columns (logicalFile, filePath and collectionName) with the external impacted files: 
+For each collection in scope of the analysis with impacted files a report is generated and written (in this case `externalImpacts_App-EPSC-main.txt`) to the `workdir`. A report will contain three columns (logicalFile, filePath and collectionName) with the external impacted files: 
 
 ```
-EPSCMORT 	 App-EPSC/cobol/epscmort.cbl 	 App-EPSC-master
+EPSCMORT   App-EPSC/cobol/epscmort.cbl   App-EPSC-main
+EPSCSMRT   App-EPSC/cobol/epscsmrt.cbl   App-EPSC-main
+EPSMLIST   App-EPSC/cobol/epsmlist.cbl   App-EPSC-main
 ```
 
+With the extended analysis turned on, the same scenario looks as below and supports a static linkage dependency chain in application App-EPSC.
+
+
+```
+...
+** Writing build list file to /u/jenkins/workspace/App-EPSM/build.20220810.122845.028/buildList.txt
+App-EPSM/cobol/epsplist.cbl
+** Perform analysis and reporting of external impacted files for the build list including changed files.
+*** Running external impact analysis with file filter **/* and collection patterns .*-main.*,.*-develop.* with analysis mode deep
+*** Running external impact analysis for files 
+     App-EPSM/copybooks-public/epsmtcom.cpy 
+     App-EPSM/cobol/epsplist.cbl 
+*** Potential external impact found EPSCSMRT (App-EPSC/cobol/epscsmrt.cbl) in collection App-EPSC-main 
+*** Potential external impact found EPSMLIST (App-EPSC/cobol/epsmlist.cbl) in collection App-EPSC-main 
+*** Potential external impact found EPSCMORT (App-EPSC/cobol/epscmort.cbl) in collection App-EPSC-main 
+**** Running external impact analysis for identified external impacted files as dependent files of the initial set. 
+     App-EPSC/cobol/epsmlist.cbl 
+     App-EPSC/cobol/epscsmrt.cbl 
+     App-EPSC/cobol/epscmort.cbl 
+**** Potential external impact found EPSMLIST (App-EPSC/link/epsmlist.lnk) in collection App-EPSC-main-outputs 
+*** Writing report of external impacts to file /u/jenkins/workspace/App-EPSM/build.20220810.122845.028/externalImpacts_App-EPSC-main-outputs.txt
+*** Writing report of external impacts to file /u/jenkins/workspace/App-EPSM/build.20220810.122845.028/externalImpacts_App-EPSC-main.txt
+** Invoking build scripts according to build order: BMS.groovy,Cobol.groovy,LinkEdit.groovy
+...
+```
 
-**Important considerations**
-- Use the pattern configuration to limit the query to those collections which are relevant and avoid unnecessary processing.
-- In most cases it is sufficient to run with the **simple** mode, which performs much faster, because it does not resolve impacts recursively. **deep** mode in fact requires searchPaths with the rules been correctly configured for the ImpactResolver API.
+**Important considerations / notes**
+- This functionality works on the assumption, that files names are unique. If the assumption is met, the results will be accurate; if not, false positives being identified. 
+- It does not perform any type of dependency resolution against search path configurations.
+- In most cases it is sufficient to run with the **simple** mode.
+- Use the pattern configuration to limit the query to those collections which are relevant and avoid unnecessary processing. Long-living branches such as _main_ and _development_ are sufficient to ran against the reporting. 
+- Use the file filter, if you can group files to be shared between applications.
 
 ## Report concurrent changes
 

diff --git a/build.groovy b/build.groovy
@@ -30,11 +30,10 @@ println("\n** Build start at $props.startTime")
 // initialize build
 initializeBuildProcess(args)
 
-// create build list and list of deletedFiles
+// create build list
 List<String> buildList = new ArrayList() 
-List<String> deletedFiles = new ArrayList()
 
-(buildList, deletedFiles) = createBuildList()
+buildList = createBuildList()
 
 // build programs in the build list
 def processCounter = 0
@@ -69,15 +68,6 @@ else {
 	}
 }
 
-// document deletions in build report
-if (deletedFiles.size() != 0 && props.documentDeleteRecords && props.documentDeleteRecords.toBoolean()) {
-	println("** Document deleted files in Build Report.")
-	if (buildUtils.assertDbbBuildToolkitVersion(props.dbbToolkitVersion, "1.1.3")) {
-		buildReportUtils = loadScript(new File("utilities/BuildReportUtilities.groovy"))
-		buildReportUtils.processDeletedFilesList(deletedFiles)
-	}
-}
-
 // finalize build process
 if (processCounter == 0)
 	processCounter = buildList.size()
@@ -457,8 +447,10 @@ def createBuildList() {
 
 	// using a set to create build list to eliminate duplicate files
 	Set<String> buildSet = new HashSet<String>()
+	Set<String> changedFiles = new HashSet<String>()
 	Set<String> deletedFiles = new HashSet<String>()
-
+	Set<String> renamedFiles = new HashSet<String>() // not yet used for any post-processing
+	Set<String> changedBuildProperties = new HashSet<String>() // not yet used for any post-processing
 	String action = (props.scanOnly) || (props.scanLoadmodules) ? 'Scanning' : 'Building'
 
 	// check if full build
@@ -470,7 +462,7 @@ def createBuildList() {
 	else if (props.impactBuild) {
 		println "** --impactBuild option selected. $action impacted programs for application ${props.application} "
 		if (repositoryClient) {
-			(buildSet, deletedFiles) = impactUtils.createImpactBuildList(repositoryClient)		}
+			(buildSet, changedFiles, deletedFiles, renamedFiles, changedBuildProperties) = impactUtils.createImpactBuildList(repositoryClient)		}
 		else {
 			println "*! Impact build requires a repository client connection to a DBB web application"
 		}
@@ -479,12 +471,12 @@ def createBuildList() {
 		println "** --mergeBuild option selected. $action changed programs for application ${props.application} flowing back to ${props.mainBuildBranch}"
 		if (repositoryClient) {
 			assert (props.topicBranchBuild) : "*! Build type --mergeBuild can only be run on for topic branch builds."
-				(buildSet, deletedFiles) = impactUtils.createMergeBuildList(repositoryClient)		}
+				(buildSet, changedFiles, deletedFiles, renamedFiles, changedBuildProperties) = impactUtils.createMergeBuildList(repositoryClient)		}
 		else {
 			println "*! Merge build requires a repository client connection to a DBB web application"
 		}
 	}
-
+	
 	// if build file present add additional files to build list (mandatory build list)
 	if (props.buildFile) {
 
@@ -514,9 +506,8 @@ def createBuildList() {
 	// now that we are done adding to the build list convert the set to a list
 	List<String> buildList = new ArrayList<String>()
 	buildList.addAll(buildSet)
-	buildSet = null
 
-	// 
+	// convert set of deleted files to a list 
 	List<String> deleteList = new ArrayList<String>()
 	deleteList.addAll(deletedFiles)
 
@@ -535,7 +526,7 @@ def createBuildList() {
 	// write out list of deleted files (for documentation, not actually used by build scripts)
 	if (deletedFiles.size() > 0){
 		String deletedFilesListLoc = "${props.buildOutDir}/deletedFilesList.${props.buildListFileExt}"
-		println "** Writing lists of deleted files to $deletedFilesListLoc"
+		println "** Writing list of deleted files to $deletedFilesListLoc"
 		File deletedFilesListFile = new File(deletedFilesListLoc)
 		deletedFilesListFile.withWriter(enc) { writer ->
 			deletedFiles.each { file ->
@@ -558,8 +549,40 @@ def createBuildList() {
 		println "** Populating file level properties from individual property files."
 		buildUtils.loadFileLevelPropertiesFromFile(buildList)
 	}
+
+	// Perform analysis and build report of external impacts
+	if (props.reportExternalImpacts && props.reportExternalImpacts.toBoolean() && repositoryClient){
+
+		if (buildUtils.assertDbbBuildToolkitVersion(props.dbbToolkitVersion, "1.1.3")) { // validate minimum dbbToolkitVersion
+			if (buildSet && changedFiles) {
+				println "** Perform analysis and reporting of external impacted files for the build list including changed files."
+				impactUtils.reportExternalImpacts(repositoryClient, buildSet.plus(changedFiles))
+			}
+			else if(buildSet) {
+				println "** Perform analysis and reporting of external impacted files for the build list."
+				impactUtils.reportExternalImpacts(repositoryClient, buildSet)
+			}
+		} else{
+			println "*! Perform analysis and reporting of external impacted files requires at least IBM Dependency Based Build Toolkit version 1.1.3."
+		}
+	}
+
+	// Document and validate concurrent changes
+	if (repositoryClient && props.reportConcurrentChanges && props.reportConcurrentChanges.toBoolean() && repositoryClient){
+		println "** Calculate and document concurrent changes."
+		impactUtils.calculateConcurrentChanges(repositoryClient, buildSet)
+	}
+
+	// document deletions in build report
+	if (deleteList.size() != 0 && props.documentDeleteRecords && props.documentDeleteRecords.toBoolean()) {
+		println("** Document deleted files in Build Report.")
+		if (buildUtils.assertDbbBuildToolkitVersion(props.dbbToolkitVersion, "1.1.3")) {
+			buildReportUtils = loadScript(new File("utilities/BuildReportUtilities.groovy"))
+			buildReportUtils.processDeletedFilesList(deleteList)
+		}
+	}
 
-	return [buildList, deleteList]
+	return buildList
 }
 
 

diff --git a/samples/application-conf/reports.properties b/samples/application-conf/reports.properties
@@ -17,8 +17,14 @@
 #
 # AnalysisDepths when performing impact analysis for external impacts
 # Options: (simple/deep)
-#   simple - will not allow recursion and only find files which have a direct dependency to the changed file 
-#   deep -  will recursively resolve impacts, which is more expensive compared to simple mode
+#
+#   simple - searches for impacted files in other collections which have a direct dependency to the changed file
+#            e.q. changed copybook COPYA which is used by cobol PGMB,
+#                 while PGMB reside in different collections
+#   deep   - will perform the simple analysis first, and pass its results back into the process to allow
+#            to find files which are impacted by the idenfitied impacted file.
+#            e.q. changed copybook COPYA which is used by cobol PGMB, which itself is referred to in linkcard LNKCRDB, 
+#                 while PGMB and LNKCARDB reside in different collections
 #
 # Default: simple
 # reportExternalImpactsAnalysisDepths=simple