Skip to content

Commit 76ede05

Browse files
jsingletjsinglet
committed
import performance suite
1 parent 878a0b2 commit 76ede05

File tree

8 files changed

+737
-0
lines changed

8 files changed

+737
-0
lines changed

‎scripts/performance_testing/Config.ps1

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
Import-Module -Name "$PSScriptRoot/../PSCodingStandards/CodingStandards"
2+
3+
$REQUIRED_CODEQL_VERSION = (Get-Content (Join-Path (Get-RepositoryRoot) "supported_codeql_configs.json") | ConvertFrom-Json).supported_environment.codeql_cli
4+

‎scripts/performance_testing/Convert-DurationStringToMs.ps1

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
function Convert-DurationStringToMs {
2+
param(
3+
[Parameter(Mandatory)]
4+
[string]
5+
$DurationString
6+
)
7+
8+
$durationStack = @()
9+
$unitStack = @()
10+
11+
12+
$durationBuff = $false
13+
$unitBuff = $false
14+
15+
for($i=0; $i -le $DurationString.Length; $i++){
16+
$s = $DurationString[$i]
17+
#Write-Host $s
18+
if($s -match "\d|\."){ # consume if it is a number or a decimal
19+
20+
# init buffer
21+
if($durationBuff -eq $false){
22+
$durationBuff = ""
23+
}
24+
25+
# accept last unit
26+
if(-Not $unitBuff -eq $false){
27+
$unitStack += $unitBuff
28+
$unitBuff = $false
29+
}
30+
31+
$durationBuff += $s
32+
}else{ # otherwise it is a unit -- multiply by it to get the ms.
33+
34+
# init buffer
35+
if($unitBuff -eq $false){
36+
$unitBuff = ""
37+
}
38+
39+
# accept last digit buffer
40+
if(-Not $durationBuff -eq $false){
41+
$durationStack += $durationBuff
42+
$durationBuff = $false
43+
}
44+
45+
$unitBuff += $s
46+
}
47+
}
48+
49+
# should always end with accepting the last one (because it will be a
50+
# unit)
51+
$unitStack += $unitBuff
52+
53+
$totalMs = 0
54+
55+
for($i=0; $i -le $unitStack.Length; $i++){
56+
57+
$time = [System.Convert]::ToDecimal($durationStack[$i])
58+
$unit = $unitStack[$i]
59+
60+
if($unit -eq 'h'){
61+
$time = $time * (60*60*1000)
62+
}
63+
if($unit -eq 'm'){
64+
$time = $time * (60*1000)
65+
}
66+
if($unit -eq 's'){
67+
$time = $time * (1000)
68+
}
69+
if($unit -eq 'ms'){
70+
$time = $time
71+
}
72+
73+
$totalMs += $time
74+
}
75+
76+
return $totalMs
77+
}

‎scripts/performance_testing/Get-DurationString.ps1

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
function Get-DurationString {
2+
param(
3+
[Parameter(Mandatory)]
4+
[string]
5+
$LogLine
6+
)
7+
$In = $LogLine.IndexOf('eval')+5
8+
$Out = $LogLine.indexof(']')
9+
10+
return $LogLine.substring($In, $Out - $In)
11+
}
12+

‎scripts/performance_testing/Get-QueryString.ps1

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
function Get-QueryString {
2+
param(
3+
[Parameter(Mandatory)]
4+
[string]
5+
$LogLine
6+
)
7+
$In = $LogLine.IndexOf('Evaluation done; writing results to ')+36
8+
$Out = $LogLine.IndexOf('.bqrs')
9+
10+
return $LogLine.SubString($In, $Out - $In)
11+
}
12+

‎scripts/performance_testing/Get-TestTmpDirectory.ps1

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
function Get-TestTmpDirectory {
2+
$Dir = [System.IO.Path]::GetTempPath()
3+
return Join-Path $Dir "$([System.Guid]::NewGuid())"
4+
}
5+

‎scripts/performance_testing/README.md

Lines changed: 211 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,211 @@
1+
# Performance Testing
2+
3+
Performance testing may be accomplished by using the performance testing tool found in this directory, `Test-ReleasePerformance.ps1`. Note that this script depends on other files from this repository. It may be run on external builds of Coding Standards through the `-CodingStandardsPath` flag, but it should be run from a fresh checkout of this repository.
4+
5+
This script requires `pwsh` to be installed. Note that the Windows native Powershell is not sufficient and you should download PowerShell Core.
6+
7+
- Installing on Windows: https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-windows?view=powershell-7.3
8+
- Installing on Linux: https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-linux?view=powershell-7.3
9+
- Installing on MacOS: https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-macos?view=powershell-7.3
10+
11+
Before invoking this script you should start a powershell session by typing `pwsh` at a command prompt.
12+
13+
## Usage
14+
15+
```
16+
NAME
17+
.\scripts\performance_testing\Test-ReleasePerformance.ps1
18+
19+
SYNOPSIS
20+
Test release performance. Generates outputs 2 csv files containing the slowest predicates as well as the queries
21+
causing work. Note that the method of computing query execution time is inaccurate due to the shared nature of
22+
predicates.
23+
24+
25+
SYNTAX
26+
C:\Projects\codeql-coding-standards\scripts\performance_testing\Test-ReleasePerformance.ps1 -RunTests [-Threads <String>] -DatabaseArchive <String>
27+
[-TestTimestamp <String>] [-CodingStandardsPath <String>] [-ResultsDirectory <String>] [-ReleaseTag <String>] -Suite <String> [-Platform <String>] -Language
28+
<String> [<CommonParameters>]
29+
30+
C:\Projects\codeql-coding-standards\scripts\performance_testing\Test-ReleasePerformance.ps1 -ProcessResults -ResultsFile <String> [-ResultsDirectory <String>]
31+
[-ReleaseTag <String>] -Suite <String> [-Platform <String>] -Language <String> [<CommonParameters>]
32+
33+
34+
DESCRIPTION
35+
Test release performance. Generates outputs 2 csv files containing the slowest predicates as well as the queries
36+
causing work. Note that the method of computing query execution time is inaccurate due to the shared nature of
37+
predicates.
38+
39+
40+
PARAMETERS
41+
-RunTests [<SwitchParameter>]
42+
Configures tool to run tests.
43+
44+
Required? true
45+
Position? named
46+
Default value False
47+
Accept pipeline input? false
48+
Accept wildcard characters? false
49+
50+
-Threads <String>
51+
Specifies the number of threads to use.
52+
53+
Required? false
54+
Position? named
55+
Default value 5
56+
Accept pipeline input? false
57+
Accept wildcard characters? false
58+
59+
-DatabaseArchive <String>
60+
Specifies the database to use for testing. Should be a zipped database
61+
directory.
62+
63+
Required? true
64+
Position? named
65+
Default value
66+
Accept pipeline input? false
67+
Accept wildcard characters? false
68+
69+
-TestTimestamp <String>
70+
The timestamp to use for the test.
71+
72+
Required? false
73+
Position? named
74+
Default value (Get-Date -Format "yyyy-MM-dd_HH-mm-ss")
75+
Accept pipeline input? false
76+
Accept wildcard characters? false
77+
78+
-CodingStandardsPath <String>
79+
The path to the coding standards root directory. This can be either the
80+
root of the repository or the root of the coding standards directory.
81+
82+
Required? false
83+
Position? named
84+
Default value "$PSScriptRoot../../"
85+
Accept pipeline input? false
86+
Accept wildcard characters? false
87+
88+
-ProcessResults [<SwitchParameter>]
89+
90+
Required? true
91+
Position? named
92+
Default value False
93+
Accept pipeline input? false
94+
Accept wildcard characters? false
95+
96+
-ResultsFile <String>
97+
Configures tool to process results.
98+
99+
Required? true
100+
Position? named
101+
Default value
102+
Accept pipeline input? false
103+
Accept wildcard characters? false
104+
105+
-ResultsDirectory <String>
106+
Where results should be written to.
107+
108+
Required? false
109+
Position? named
110+
Default value (Get-Location)
111+
Accept pipeline input? false
112+
Accept wildcard characters? false
113+
114+
-ReleaseTag <String>
115+
The release tag to use for the test.
116+
117+
Required? false
118+
Position? named
119+
Default value current
120+
Accept pipeline input? false
121+
Accept wildcard characters? false
122+
123+
-Suite <String>
124+
Which suite to run.
125+
126+
Required? true
127+
Position? named
128+
Default value
129+
Accept pipeline input? false
130+
Accept wildcard characters? false
131+
132+
-Platform <String>
133+
The platform to run on. This is just a descriptive string.
134+
135+
Required? false
136+
Position? named
137+
Default value $PSVersionTable.Platform
138+
Accept pipeline input? false
139+
Accept wildcard characters? false
140+
141+
-Language <String>
142+
The language to run on.
143+
144+
Required? true
145+
Position? named
146+
Default value
147+
Accept pipeline input? false
148+
Accept wildcard characters? false
149+
150+
<CommonParameters>
151+
This cmdlet supports the common parameters: Verbose, Debug,
152+
ErrorAction, ErrorVariable, WarningAction, WarningVariable,
153+
OutBuffer, PipelineVariable, and OutVariable. For more information, see
154+
about_CommonParameters (https://go.microsoft.com/fwlink/?LinkID=113216).
155+
156+
INPUTS
157+
158+
OUTPUTS
159+
160+
161+
RELATED LINKS
162+
163+
```
164+
## Example Usage
165+
166+
Run the `cert` suite for `c` from within the Coding Standards repository.
167+
168+
```
169+
.\scripts\performance_testing\Test-ReleasePerformance.ps1 -RunTests -DatabaseArchive ..\codeql-coding-standards-release-engineering\data\commaai-openpilot-72d1744d830bc249d8761a1d843a98fb0ced49fe-cpp.zip -Suite cert -Language c
170+
```
171+
172+
Run the `cert` suite for `c` on an external release, specifying a `-ReleaseTag` as well. The `-ReleaseTag` parameter does not have to match the code you are testing, it is for organization purposes only.
173+
174+
```
175+
.\scripts\performance_testing\Test-ReleasePerformance.ps1 -RunTests -DatabaseArchive ..\codeql-coding-standards-release-engineering\data\commaai-openpilot-72d1744d830bc249d8761a1d843a98fb0ced49fe-cpp.zip -Suite cert -Language c -ReleaseTag "2.16.0" -CodingStandardsPath "Downloads\code-scanning-cpp-query-pack-2.16.0\codeql-coding-standards\"
176+
```
177+
178+
179+
180+
## Outputs
181+
182+
The `Test-ReleasePerformance.ps1` produces three files in the `ResultsDirectory` location, which defaults `performance_tests` within the current working directory.
183+
184+
- `suite=$Suite,datum=queries.csv` - Which contains the run time for each query.
185+
- `suite=$Suite,datum=evaluator-log.json` - Which contains the evaluator log.
186+
- `suite=$Suite,datum=sarif.sarif` - The sarif log file for the run.
187+
188+
## Profiling Predicates
189+
190+
If you wish to extract predicate-level profiling information, you may use the script `profile_predicates.py` located in this directory. It requires Python3 with `pandas` and `numpy` to work. If you wish to use a virtual environment you may create one as follows on a Unix-based platform:
191+
192+
```
193+
python -mvenv venv
194+
source venv/bin/activate
195+
pip install pandas numpy
196+
```
197+
198+
The script works by summarizing ALL of the csv and json files within a given directory. Thus, if you want to profile multiple suites or multiple releases you may place the files within that directory by repeatedly invoking `Test-ReleasePerformance.ps1.` Make sure to supply the same output directory each time so that the results accumulate in the correct location.
199+
200+
To invoke the script run:
201+
202+
```
203+
python scripts/performance_testing/profile_predicates.py <path to output directory>
204+
```
205+
206+
For example:
207+
```
208+
python .\scripts\performance_testing\profile_predicates.py .\performance_tests\
209+
```
210+
211+
This will produce an additional CSV file per release, platform, and language within that directory called: `slow-log,datum=predicates,release={release},platform={platform},language={language}.csv` which will contain the execution times of all of the predicates used during execution.

0 commit comments

Comments
 (0)