Page Scripting test result visualizer & updated test analyzer logic. #1423
Conversation
|
This can't be fully tested yet, due to a current dependency error in bc-replay which fails on install. |
| [string] $project = '.' | ||
| [string] $project = '.', | ||
| [Parameter(HelpMessage = "Tests to analyze", Mandatory = $false)] | ||
| [string] $testsToAnalyze = 'app' |
Actions/AnalyzeTests/README.md
Outdated
| @@ -14,6 +14,7 @@ none | |||
| | :-- | :-: | :-- | :-- | | |||
| | shell | | The shell (powershell or pwsh) in which the PowerShell script in this action should run | powershell | | |||
| | project | Yes | Name of project to analyze or . if the repository is setup for single project | | | |||
| | testsToAnalyze | No | Which set of tests to analyze. Should be one of ('app', 'bcpt', 'pageScripting') | app | | |||
There was a problem hiding this comment.
Minor: how about testType with possible values "normal" (or "test"), "bcpt" and "pageScripting".
| -baseLinePath $bcptBaseLineFile ` | ||
| -thresholdsPath $bcptThresholdsFile ` | ||
| -bcptThresholds ($settings.bcptThresholds | ConvertTo-HashTable) | ||
| if ($testsToAnalyze -eq 'app') { |
There was a problem hiding this comment.
Consider using a switch statement instead.
|
|
||
| - name: Analyze BCPT Test Results | ||
| id: analyzeTestResultsBCPT | ||
| if: (success() || failure()) && env.doNotRunBcptTests == 'False' |
There was a problem hiding this comment.
Is this env variable set?
I believe it needs to be included in "Read Settings", line 97.
Same for doNotRunpageScriptingTests.
| @@ -14,6 +14,7 @@ none | |||
| | :-- | :-: | :-- | :-- | | |||
| | shell | | The shell (powershell or pwsh) in which the PowerShell script in this action should run | powershell | | |||
| | project | Yes | Name of project to analyze or . if the repository is setup for single project | | | |||
| | testsToAnalyze | No | Which set of tests to analyze. Should be one of ('app', 'bcpt', 'pageScripting') | app | | |||
|
|
|||
| ## OUTPUT | |||
There was a problem hiding this comment.
I don't think the action has output. OUTPUT variables might need to be cleaned up.
|
|
||
| #Write total summary for all test suites | ||
| Write-Host "$totalTests tests, $totalPassed passed, $totalFailed failed, $totalSkipped skipped, $totalTime seconds" | ||
| $summarySb.AppendLine('|Suite|Tests|Passed|Failed|Skipped|Time|') | Out-Null |
There was a problem hiding this comment.
PowerShell generating Markdown is extremely hard to read.
Is there a way to define some MD templates with placeholders that could be used to generate the end-goal MD?
There was a problem hiding this comment.
I'll try see if I can come up with something that is easier to debug / code review.
I added functionality to visualize page scripting tests, similar to how app and bcpt tests are visualized as MD.
I also updated the test analyzer action to take a new param, which decides which of those 3 types of tests to analyze and visualize.
I think this is a better approach that a separate action for analyzing page scripting tests, as it is nearly identical in implementation.
This approach also splits the analyzer visualization step into 3 separate steps, for each of the 3 types. This have the advantage, that we can easily add new test types to the analyzer in the future, but we are also more likely to display the results in the build summary. As each step has a limit of 1MiB, the current approach of displaying both app and bcpt results in the same step, are more likely to reach that limit. Now each summary for each type have a separate limit of 1MiB.