Skip to content

Add azure-typespec-author benchmark ADO pipeline #15516

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
haolingdong-msft wants to merge 103 commits into
base: main
Choose a base branch
from
Draft

Add azure-typespec-author benchmark ADO pipeline #15516

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
103 commits
Select commit Hold shift + click to select a range
1131e61
Move existing benchmark cases from waza to evaluate
JoyerJin Apr 17, 2026
40e0b08
update verify plan and use sdk mcp config
JoyerJin Apr 20, 2026
a47d155
Update .vally.yaml
JoyerJin Apr 20, 2026
9ba5efc
Update 005001.eval.yaml
JoyerJin Apr 20, 2026
e537a9a
Merge branch 'main' into joyer/benchmark-test-tools-repo
JoyerJin Apr 21, 2026
daca67e
add tspconfig.yaml for 003001 and 004003, edit tools name, add all.ev…
JoyerJin Apr 21, 2026
d3ea768
update cases
JoyerJin Apr 22, 2026
1aad636
remove 003001 and reorder 003-LRO, edit eval files to improve quality
JoyerJin Apr 23, 2026
3fe9bf7
[azure-typespec-author skill] Consolidate reference documents into on…
haolingdong-msft Apr 24, 2026
4b293e3
update workflow highlights
haolingdong-msft Apr 24, 2026
79bca6b
update more case graders
JoyerJin Apr 24, 2026
1f8cb59
add skill invocation
JoyerJin Apr 27, 2026
7098b9e
Add back mcp tools and constrain sections
haolingdong-msft Apr 27, 2026
fb12e77
resolve comments
haolingdong-msft Apr 27, 2026
a3c8b1f
edit package
JoyerJin Apr 27, 2026
78a8781
Add azure-typespec-author skill evaluation pipeline
haolingdong-msft Apr 28, 2026
a8bac91
Merge remote-tracking branch 'origin/remove-task-type-classification'…
haolingdong-msft Apr 28, 2026
9adccdc
Merge remote-tracking branch 'origin/joyer/benchmark-test-tools-repo'…
haolingdong-msft Apr 28, 2026
e3af6a8
add 5 test cases
haolingdong-msft Apr 28, 2026
ed9385a
Fix CI: log service output and improve startup health check
haolingdong-msft Apr 29, 2026
54f67b2
Add Azure SDK QA Bot Dev Variables group to pipeline
haolingdong-msft Apr 29, 2026
792e0c3
Fix npm auth: use azsdk-microsoft-package-github-token for GitHub Pac…
haolingdong-msft Apr 29, 2026
8165d99
Update evaluate stage: add Copilot SDK install and GITHUB_TOKEN for v…
haolingdong-msft Apr 30, 2026
afb2017
Fix npm install: use local prefix to avoid EPERM on global install
haolingdong-msft Apr 30, 2026
89731b1
Fix PATH: use bash \C:\Users\haolingdong not ADO \ for npm-global bin
haolingdong-msft Apr 30, 2026
4c4172d
Switch to ubuntu-latest pool to fix registry.npmjs.org EPERM on 1ES m…
haolingdong-msft Apr 30, 2026
e5f3365
Update evaluate stage to match jeo02 reference: tag-based eval, simpl…
haolingdong-msft Apr 30, 2026
2b37fd8
Remove 1ES PT wrapper and use azsdk-pool directly to allow npm regist…
haolingdong-msft May 6, 2026
03e6cd2
Use Azure Artifacts npm mirror for 1ES pool egress
haolingdong-msft May 6, 2026
c5ba7a9
Use Use1ESOfficial: true to satisfy 1ES PT drift check on internal pr…
haolingdong-msft May 6, 2026
020bed2
Authenticate npm against Azure Artifacts feed via create-authenticate…
haolingdong-msft May 6, 2026
1abf9cd
Run vally from azure-typespec-author/evaluate so it finds .vally.yaml…
haolingdong-msft May 6, 2026
3d8c2d5
Fix eval 001002.eval.yaml: timeout duration string + forward-slash paths
haolingdong-msft May 6, 2026
f2c93a8
Fix eval 001001.eval.yaml: timeout duration string + forward-slash paths
haolingdong-msft May 6, 2026
9438cb5
Fix eval 001003.eval.yaml: timeout duration string + forward-slash paths
haolingdong-msft May 6, 2026
de2223e
Fix eval 001005.eval.yaml: timeout duration string + forward-slash paths
haolingdong-msft May 6, 2026
5151caa
Fix eval 001004.eval.yaml: timeout duration string + forward-slash paths
haolingdong-msft May 6, 2026
21fde59
Fix .vally.yaml mcp timeout to duration string
haolingdong-msft May 6, 2026
5d84f2d
Set MCP server timeout to 30m
haolingdong-msft May 6, 2026
a1319a1
Merge remote-tracking branch 'origin/main' into add-pipeline
haolingdong-msft May 6, 2026
4591dcb
update skill
haolingdong-msft May 6, 2026
a87cc7a
Merge remote-tracking branch 'origin/add-pipeline' into add-pipeline
haolingdong-msft May 6, 2026
44bc8ae
fix merge issue
haolingdong-msft May 6, 2026
3881161
Revert 001001.eval.yaml timeout to 1800
haolingdong-msft May 6, 2026
d787db7
Revert 001002.eval.yaml timeout to 1800
haolingdong-msft May 6, 2026
145be2b
Revert 001003.eval.yaml timeout to 1800
haolingdong-msft May 6, 2026
1f2f97d
Revert 001004.eval.yaml timeout to 1800
haolingdong-msft May 6, 2026
6779354
Revert 001005.eval.yaml timeout to 1800
haolingdong-msft May 6, 2026
cddb272
Revert 001001.eval.yaml to match main (PR #15516)
haolingdong-msft May 6, 2026
2a44e11
Revert 001002.eval.yaml to match main (PR #15516)
haolingdong-msft May 6, 2026
fa5060e
Revert 001003.eval.yaml to match main (PR #15516)
haolingdong-msft May 6, 2026
cbfd605
Revert 001004.eval.yaml to match main (PR #15516)
haolingdong-msft May 6, 2026
f59c46b
Revert 001005.eval.yaml to match main (PR #15516)
haolingdong-msft May 6, 2026
f435731
Pipeline-side normalize bare timeout to duration string for vally
haolingdong-msft May 6, 2026
72d9587
Rewrite pipeline yml cleanly (LF, no mid-word CRLFs); add timeout nor…
haolingdong-msft May 6, 2026
22cf3c8
Fix sed regex to handle inline comments after timeout
haolingdong-msft May 6, 2026
737b3e9
Set timeout to '1800s' in 001001.eval.yaml
haolingdong-msft May 6, 2026
ae287d2
Set timeout to '1800s' in 001002.eval.yaml
haolingdong-msft May 6, 2026
0aa6062
Set timeout to '1800s' in 001003.eval.yaml
haolingdong-msft May 6, 2026
ddea51a
Set timeout to '1800s' in 001004.eval.yaml
haolingdong-msft May 6, 2026
a29165e
Set timeout to '1800s' in 001005.eval.yaml
haolingdong-msft May 6, 2026
b5949b6
Set timeout to '1800s' in 001006.eval.yaml
haolingdong-msft May 6, 2026
0384381
Set timeout to '1800s' in 001007.eval.yaml
haolingdong-msft May 6, 2026
2d73b24
Set timeout to '1800s' in 001008.eval.yaml
haolingdong-msft May 6, 2026
c1cd20f
Set timeout to '1800s' in 001009.eval.yaml
haolingdong-msft May 6, 2026
3a309f4
Set timeout to '1800s' in 001010.eval.yaml
haolingdong-msft May 6, 2026
ef9b120
Set timeout to '1800s' in 001011.eval.yaml
haolingdong-msft May 6, 2026
e62f76f
Set timeout to '1800s' in 001012.eval.yaml
haolingdong-msft May 6, 2026
29a1a99
Set timeout to '1800s' in 001013.eval.yaml
haolingdong-msft May 6, 2026
6a11d44
Set timeout to '1800s' in 002001.eval.yaml
haolingdong-msft May 6, 2026
834e558
Set timeout to '1800s' in 002002.eval.yaml
haolingdong-msft May 6, 2026
b2bfb0e
Set timeout to '1800s' in 002003.eval.yaml
haolingdong-msft May 6, 2026
55c8ab7
Set timeout to '1800s' in 002004.eval.yaml
haolingdong-msft May 6, 2026
ecc5b1c
Set timeout to '1800s' in 002005.eval.yaml
haolingdong-msft May 6, 2026
8ce669a
Set timeout to '1800s' in 002006.eval.yaml
haolingdong-msft May 6, 2026
e783244
Set timeout to '1800s' in 002007.eval.yaml
haolingdong-msft May 6, 2026
5fac5e1
Set timeout to '1800s' in 002008.eval.yaml
haolingdong-msft May 6, 2026
97afda4
Set timeout to '1800s' in 002009.eval.yaml
haolingdong-msft May 6, 2026
d6e10cc
Set timeout to '1800s' in 002010.eval.yaml
haolingdong-msft May 6, 2026
f6d1225
Set timeout to '1800s' in 003001.eval.yaml
haolingdong-msft May 6, 2026
3147320
Set timeout to '1800s' in 003002.eval.yaml
haolingdong-msft May 6, 2026
4f6d14b
Set timeout to '1800s' in 004001.eval.yaml
haolingdong-msft May 6, 2026
11d213d
Set timeout to '1800s' in 004002.eval.yaml
haolingdong-msft May 6, 2026
67aa27b
Set timeout to '1800s' in 004003.eval.yaml
haolingdong-msft May 6, 2026
759a211
Set timeout to '1800s' in 005001.eval.yaml
haolingdong-msft May 6, 2026
659499b
Add SkillBranch parameter to azure-typespec-author benchmark pipeline
Copilot May 6, 2026
8305eda
Revert skill .md changes; default SkillBranch to remove-task-type-cla…
Copilot May 6, 2026
09a23e3
Set MCP server timeout to '30m' (was bare int 300000)
haolingdong-msft May 6, 2026
0609476
Use trigger branch by default for SkillBranch (was hardcoded to 'remo…
haolingdong-msft May 6, 2026
9a44da6
Re-add SkillBranch param; overlay only skill source (preserve evaluate/)
Copilot May 6, 2026
d3c0d7a
Make SkillBranch a dropdown of selectable branches
Copilot May 6, 2026
492b2c1
Revert "Make SkillBranch a dropdown of selectable branches"
Copilot May 6, 2026
886c459
Reduce timeout from 30 minutes to 10 minutes
haolingdong-msft May 6, 2026
526bf22
Update .vally.yaml
haolingdong-msft May 6, 2026
5247d1d
Add verbose flag to vally eval command
haolingdong-msft May 6, 2026
5639d95
Pre-build MCP server, auth NuGet, install typespec, smoke test
haolingdong-msft May 6, 2026
fc7889c
Install .NET SDK from global.json (9.0.x)
haolingdong-msft May 6, 2026
8c60517
Make MCP smoke test informational (list tools, dont fail)
haolingdong-msft May 6, 2026
45f176f
install vally 0.4.0
JoyerJin May 7, 2026
1aa72b3
edit fixture path for 005-008
JoyerJin May 7, 2026
f186bb8
change timeout and add testing with workers running
JoyerJin May 7, 2026
0decb7c
Update display name for evaluation step in pipeline
JoyerJin May 7, 2026
0eb2a55
Update second benchmark of pipeline for error handling
JoyerJin May 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/skills/azure-typespec-author/evaluate/.vally.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ environments:
type: stdio
command: dotnet
args: ["run", "--project", "../../../../tools/azsdk-cli/Azure.Sdk.Tools.Cli", "--", "start"]
timeout: 300000
timeout: "10m"
env:
AZSDKTOOLS_AGENT_TESTING: "false"
AZSDKTOOLS_COLLECT_TELEMETRY: "false"
Expand All @@ -25,7 +25,7 @@ environments:
type: stdio
command: dotnet
args: ["run", "--project", "../../../../tools/azsdk-cli/Azure.Sdk.Tools.Cli", "--", "start"]
timeout: 300000
timeout: "10m"
env:
AZSDKTOOLS_AGENT_TESTING: "false"
AZSDKTOOLS_COLLECT_TELEMETRY: "false"
Expand All @@ -44,4 +44,4 @@ suites:
warning:
evals: ["evals/005001.eval.yaml"]
all:
evals: ["evals/*.eval.yaml"]
evals: ["evals/*.eval.yaml"]
8 changes: 4 additions & 4 deletions .github/skills/azure-typespec-author/evaluate/evals/001001.eval.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,10 @@ environment: azsdk-mcp

# Execution configuration
config:
runs: 1 # Trials per stimulus
timeout: 1800 # Seconds per trial
model: claude-opus-4.6 #gpt-4o #claude-sonnet-4.6 # Model for agent execution
executor: copilot-sdk # Which executor to use
runs: 1 # Trials per stimulus
timeout: "660s" # Seconds per trial
model: claude-opus-4.6 #gpt-4o #claude-sonnet-4.6 # Model for agent execution
executor: copilot-sdk # Which executor to use

# Test cases
stimuli:
Expand Down
8 changes: 4 additions & 4 deletions .github/skills/azure-typespec-author/evaluate/evals/001002.eval.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,10 @@ environment: azsdk-mcp

# Execution configuration
config:
runs: 1 # Trials per stimulus
timeout: 1800 # Seconds per trial
model: claude-opus-4.6 #gpt-4o #claude-sonnet-4.6 # Model for agent execution
executor: copilot-sdk # Which executor to use
runs: 1 # Trials per stimulus
timeout: "660s" # Seconds per trial
model: claude-opus-4.6 #gpt-4o #claude-sonnet-4.6 # Model for agent execution
executor: copilot-sdk # Which executor to use

# Test cases
stimuli:
Expand Down
8 changes: 4 additions & 4 deletions .github/skills/azure-typespec-author/evaluate/evals/001003.eval.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,10 @@ environment: azsdk-mcp

# Execution configuration
config:
runs: 1 # Trials per stimulus
timeout: 1800 # Seconds per trial
model: claude-opus-4.6 #gpt-4o #claude-sonnet-4.6 # Model for agent execution
executor: copilot-sdk # Which executor to use
runs: 1 # Trials per stimulus
timeout: "660s" # Seconds per trial
model: claude-opus-4.6 #gpt-4o #claude-sonnet-4.6 # Model for agent execution
executor: copilot-sdk # Which executor to use

# Test cases
stimuli:
Expand Down
8 changes: 4 additions & 4 deletions .github/skills/azure-typespec-author/evaluate/evals/001004.eval.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,10 @@ environment: azsdk-mcp

# Execution configuration
config:
runs: 1 # Trials per stimulus
timeout: 1800 # Seconds per trial
model: claude-opus-4.6 #gpt-4o #claude-sonnet-4.6 # Model for agent execution
executor: copilot-sdk # Which executor to use
runs: 1 # Trials per stimulus
timeout: "660s" # Seconds per trial
model: claude-opus-4.6 #gpt-4o #claude-sonnet-4.6 # Model for agent execution
executor: copilot-sdk # Which executor to use

# Test cases
stimuli:
Expand Down
55 changes: 28 additions & 27 deletions .github/skills/azure-typespec-author/evaluate/evals/001005.eval.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,10 @@ environment: azsdk-mcp

# Execution configuration
config:
runs: 1 # Trials per stimulus
timeout: 1800 # Seconds per trial
model: claude-opus-4.6 #gpt-4o #claude-sonnet-4.6 # Model for agent execution
executor: copilot-sdk # Which executor to use
runs: 1 # Trials per stimulus
timeout: "660s" # Seconds per trial
model: claude-opus-4.6 #gpt-4o #claude-sonnet-4.6 # Model for agent execution
executor: copilot-sdk # Which executor to use

# Test cases
stimuli:
Expand All @@ -23,57 +23,58 @@ stimuli:
max_tokens: 5000
environment:
files:
- src: ../fixtures/001005-version-add-preview-after-preview\employee.tsp
- src: ../fixtures/001005-version-add-preview-after-preview/employee.tsp
dest: employee.tsp
- src: ../fixtures/001005-version-add-preview-after-preview\main.tsp
- src: ../fixtures/001005-version-add-preview-after-preview/main.tsp
dest: main.tsp
- src: ../fixtures/001005-version-add-preview-after-preview\shared.tsp
- src: ../fixtures/001005-version-add-preview-after-preview/shared.tsp
dest: shared.tsp
- src: ../fixtures/001005-version-add-preview-after-preview\tspconfig.yaml
- src: ../fixtures/001005-version-add-preview-after-preview/tspconfig.yaml
dest: tspconfig.yaml
- src: ../fixtures/001005-version-add-preview-after-preview\package.json
- src: ../fixtures/001005-version-add-preview-after-preview/package.json
dest: package.json
- src: ../fixtures/001005-version-add-preview-after-preview\examples\2024-10-01-preview\Employees_CreateOrUpdate_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview/examples/2024-10-01-preview/Employees_CreateOrUpdate_MaximumSet_Gen.json
dest: examples/2024-10-01-preview/Employees_CreateOrUpdate_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview\examples\2024-10-01-preview\Employees_Delete_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview/examples/2024-10-01-preview/Employees_Delete_MaximumSet_Gen.json
dest: examples/2024-10-01-preview/Employees_Delete_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview\examples\2024-10-01-preview\Employees_Get_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview/examples/2024-10-01-preview/Employees_Get_MaximumSet_Gen.json
dest: examples/2024-10-01-preview/Employees_Get_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview\examples\2024-10-01-preview\Employees_ListByResourceGroup_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview/examples/2024-10-01-preview/Employees_ListByResourceGroup_MaximumSet_Gen.json
dest: examples/2024-10-01-preview/Employees_ListByResourceGroup_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview\examples\2024-10-01-preview\Employees_ListBySubscription_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview/examples/2024-10-01-preview/Employees_ListBySubscription_MaximumSet_Gen.json
dest: examples/2024-10-01-preview/Employees_ListBySubscription_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview\examples\2024-10-01-preview\Employees_Update_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview/examples/2024-10-01-preview/Employees_Update_MaximumSet_Gen.json
dest: examples/2024-10-01-preview/Employees_Update_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview\examples\2024-10-01-preview\Operations_List_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview/examples/2024-10-01-preview/Operations_List_MaximumSet_Gen.json
dest: examples/2024-10-01-preview/Operations_List_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview\examples\2024-10-01-preview\Operations_List_MinimumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview/examples/2024-10-01-preview/Operations_List_MinimumSet_Gen.json
dest: examples/2024-10-01-preview/Operations_List_MinimumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview\examples\2021-10-01\Employees_CreateOrUpdate_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview/examples/2021-10-01/Employees_CreateOrUpdate_MaximumSet_Gen.json
dest: examples/2021-10-01/Employees_CreateOrUpdate_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview\examples\2021-10-01\Employees_Delete_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview/examples/2021-10-01/Employees_Delete_MaximumSet_Gen.json
dest: examples/2021-10-01/Employees_Delete_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview\examples\2021-10-01\Employees_Get_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview/examples/2021-10-01/Employees_Get_MaximumSet_Gen.json
dest: examples/2021-10-01/Employees_Get_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview\examples\2021-10-01\Employees_ListByResourceGroup_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview/examples/2021-10-01/Employees_ListByResourceGroup_MaximumSet_Gen.json
dest: examples/2021-10-01/Employees_ListByResourceGroup_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview\examples\2021-10-01\Employees_ListByResourceGroup_MinimumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview/examples/2021-10-01/Employees_ListByResourceGroup_MinimumSet_Gen.json
dest: examples/2021-10-01/Employees_ListByResourceGroup_MinimumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview\examples\2021-10-01\Employees_ListBySubscription_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview/examples/2021-10-01/Employees_ListBySubscription_MaximumSet_Gen.json
dest: examples/2021-10-01/Employees_ListBySubscription_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview\examples\2021-10-01\Employees_ListBySubscription_MinimumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview/examples/2021-10-01/Employees_ListBySubscription_MinimumSet_Gen.json
dest: examples/2021-10-01/Employees_ListBySubscription_MinimumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview\examples\2021-10-01\Employees_Update_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview/examples/2021-10-01/Employees_Update_MaximumSet_Gen.json
dest: examples/2021-10-01/Employees_Update_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview\examples\2021-10-01\Operations_List_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview/examples/2021-10-01/Operations_List_MaximumSet_Gen.json
dest: examples/2021-10-01/Operations_List_MaximumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview\examples\2021-10-01\Operations_List_MinimumSet_Gen.json
- src: ../fixtures/001005-version-add-preview-after-preview/examples/2021-10-01/Operations_List_MinimumSet_Gen.json
dest: examples/2021-10-01/Operations_List_MinimumSet_Gen.json
graders:
- type: tool-calls
config:
required:
- edit
- web_fetch
- azure-sdk-mcp-azsdk_run_typespec_validation
- type: skill-invocation
config:
Expand Down
55 changes: 28 additions & 27 deletions .github/skills/azure-typespec-author/evaluate/evals/001006.eval.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,10 @@ environment: azsdk-mcp

# Execution configuration
config:
runs: 1 # Trials per stimulus
timeout: 1800 # Seconds per trial
model: claude-opus-4.6 #gpt-4o #claude-sonnet-4.6 # Model for agent execution
executor: copilot-sdk # Which executor to use
runs: 1 # Trials per stimulus
timeout: "660s" # Seconds per trial
model: claude-opus-4.6 #gpt-4o #claude-sonnet-4.6 # Model for agent execution
executor: copilot-sdk # Which executor to use

# Test cases
stimuli:
Expand All @@ -23,57 +23,58 @@ stimuli:
max_tokens: 5000
environment:
files:
- src: ../fixtures/001006-version-add-preview-after-stable\employee.tsp
- src: ../fixtures/001006-version-add-preview-after-stable/employee.tsp
dest: employee.tsp
- src: ../fixtures/001006-version-add-preview-after-stable\main.tsp
- src: ../fixtures/001006-version-add-preview-after-stable/main.tsp
dest: main.tsp
- src: ../fixtures/001006-version-add-preview-after-stable\shared.tsp
- src: ../fixtures/001006-version-add-preview-after-stable/shared.tsp
dest: shared.tsp
- src: ../fixtures/001006-version-add-preview-after-stable\tspconfig.yaml
- src: ../fixtures/001006-version-add-preview-after-stable/tspconfig.yaml
dest: tspconfig.yaml
- src: ../fixtures/001006-version-add-preview-after-stable\package.json
- src: ../fixtures/001006-version-add-preview-after-stable/package.json
dest: package.json
- src: ../fixtures/001006-version-add-preview-after-stable\examples\2021-10-01\Employees_CreateOrUpdate_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable/examples/2021-10-01/Employees_CreateOrUpdate_MaximumSet_Gen.json
dest: examples/2021-10-01/Employees_CreateOrUpdate_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable\examples\2021-10-01\Employees_Delete_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable/examples/2021-10-01/Employees_Delete_MaximumSet_Gen.json
dest: examples/2021-10-01/Employees_Delete_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable\examples\2021-10-01\Employees_Get_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable/examples/2021-10-01/Employees_Get_MaximumSet_Gen.json
dest: examples/2021-10-01/Employees_Get_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable\examples\2021-10-01\Employees_ListByResourceGroup_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable/examples/2021-10-01/Employees_ListByResourceGroup_MaximumSet_Gen.json
dest: examples/2021-10-01/Employees_ListByResourceGroup_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable\examples\2021-10-01\Employees_ListBySubscription_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable/examples/2021-10-01/Employees_ListBySubscription_MaximumSet_Gen.json
dest: examples/2021-10-01/Employees_ListBySubscription_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable\examples\2021-10-01\Employees_Update_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable/examples/2021-10-01/Employees_Update_MaximumSet_Gen.json
dest: examples/2021-10-01/Employees_Update_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable\examples\2021-10-01\Operations_List_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable/examples/2021-10-01/Operations_List_MaximumSet_Gen.json
dest: examples/2021-10-01/Operations_List_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable\examples\2021-10-01\Operations_List_MinimumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable/examples/2021-10-01/Operations_List_MinimumSet_Gen.json
dest: examples/2021-10-01/Operations_List_MinimumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable\examples\2024-10-01\Employees_CreateOrUpdate_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable/examples/2024-10-01/Employees_CreateOrUpdate_MaximumSet_Gen.json
dest: examples/2024-10-01/Employees_CreateOrUpdate_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable\examples\2024-10-01\Employees_Delete_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable/examples/2024-10-01/Employees_Delete_MaximumSet_Gen.json
dest: examples/2024-10-01/Employees_Delete_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable\examples\2024-10-01\Employees_Get_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable/examples/2024-10-01/Employees_Get_MaximumSet_Gen.json
dest: examples/2024-10-01/Employees_Get_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable\examples\2024-10-01\Employees_ListByResourceGroup_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable/examples/2024-10-01/Employees_ListByResourceGroup_MaximumSet_Gen.json
dest: examples/2024-10-01/Employees_ListByResourceGroup_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable\examples\2024-10-01\Employees_ListByResourceGroup_MinimumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable/examples/2024-10-01/Employees_ListByResourceGroup_MinimumSet_Gen.json
dest: examples/2024-10-01/Employees_ListByResourceGroup_MinimumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable\examples\2024-10-01\Employees_ListBySubscription_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable/examples/2024-10-01/Employees_ListBySubscription_MaximumSet_Gen.json
dest: examples/2024-10-01/Employees_ListBySubscription_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable\examples\2024-10-01\Employees_ListBySubscription_MinimumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable/examples/2024-10-01/Employees_ListBySubscription_MinimumSet_Gen.json
dest: examples/2024-10-01/Employees_ListBySubscription_MinimumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable\examples\2024-10-01\Employees_Update_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable/examples/2024-10-01/Employees_Update_MaximumSet_Gen.json
dest: examples/2024-10-01/Employees_Update_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable\examples\2024-10-01\Operations_List_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable/examples/2024-10-01/Operations_List_MaximumSet_Gen.json
dest: examples/2024-10-01/Operations_List_MaximumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable\examples\2024-10-01\Operations_List_MinimumSet_Gen.json
- src: ../fixtures/001006-version-add-preview-after-stable/examples/2024-10-01/Operations_List_MinimumSet_Gen.json
dest: examples/2024-10-01/Operations_List_MinimumSet_Gen.json
graders:
- type: tool-calls
config:
required:
- edit
- web_fetch
- azure-sdk-mcp-azsdk_run_typespec_validation
- type: skill-invocation
config:
Expand Down
Loading
Loading