Skip to content

Split commands.yml into running and saving #18688

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
289 changes: 170 additions & 119 deletions .github/workflows/commands.yml
Original file line number Diff line number Diff line change
@@ -1,131 +1,182 @@
name: Commands on PR
name: Run CLI Commands via PR Comment

on:
issue_comment:
types: [created]
schedule:
# once a day at 13:00 UTC
- cron: '0 13 * * *'

permissions:
contents: write
issues: write
pull-requests: read

jobs:
cleanup_old_runs:
if: github.event.schedule == '0 13 * * *'
# This first job by definiton runs user-supplied code - you must NOT elevate its permissions to `write`
Comment on lines -15 to -16
Copy link
Member

@vzarytovskii vzarytovskii Jun 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was used to cleanup workflow runs so they don't pollute the log, will they just remain there forever?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GH Actions have their own cleanup, configured via project settings. I would leave it to that.

# Malicious code could change nuget source URL, build targets or even compiler itself to pass a GH token
# And use it to create branches, spam issues etc. Any write-actions happen in the second job, which does not allow
# user extension points (i.e. plain scripts, must NOT run scripts from within checked-out code)
detect-and-run:
runs-on: ubuntu-latest
permissions:
actions: write
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
outputs:
command: ${{ steps.parse.outputs.command }}
arg: ${{ steps.parse.outputs.arguments }}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: Crucial to verify that this command is always from the known set.
And gets not set if it is something else ( = the comment-pipeline must fail on unknown input0

if: github.event.issue.pull_request
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Crucial to verify:
This must be used in a safe way, especially in the second job.

steps:
- name: Delete old workflow runs
run: |
_UrlPath="/repos/$GITHUB_REPOSITORY/actions/workflows"
_CurrentWorkflowID="$(gh api -X GET "$_UrlPath" | jq '.workflows[] | select(.name == '\""$GITHUB_WORKFLOW"\"') | .id')"
- name: Parse comment
id: parse
uses: dotnet/comment-pipeline@1
with:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using a dotnet fork of the pipeline instead of the original.

comment: ${{ toJSON(github.event.comment) }}
commands: |
/run fantomas
/run ilverify
/run xlf
/run test-baseline
github-token: ${{ secrets.GITHUB_TOKEN }}

# delete workitems which are 'completed'. (other candidate values of status field are: 'queued' and 'in_progress')
- name: Checkout the repository
uses: actions/checkout@v4

- name: Checkout PR branch
if: ${{ steps.parse.outputs.command }}
run: gh auth setup-git && gh pr checkout ${{ github.event.issue.number }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

gh api -X GET "$_UrlPath/$_CurrentWorkflowID/runs" --paginate \
| jq '.workflow_runs[] | select(.status == "completed") | .id' \
| xargs -I{} gh api -X DELETE "/repos/$GITHUB_REPOSITORY/actions/runs"/{}
- name: Install dotnet
uses: actions/setup-dotnet@v3
with:
global-json-file: global.json

- name: Install dotnet tools
run: dotnet tool restore

- name: Setup .NET 9.0.0 Runtime for test execution
if: ${{ steps.parse.outputs.command == '/run test-baseline' }}
uses: actions/setup-dotnet@v4
with:
dotnet-version: '9.0.x'

run_command:
if: github.event.issue.pull_request != '' && contains(github.event.comment.body, '/run')
- name: Run command
id: run-cmd
env:
TEST_UPDATE_BSL: 1
continue-on-error: true
run: |
case "${{ steps.parse.outputs.command }}" in
"/run fantomas") dotnet fantomas . ;;
"/run xlf") dotnet build src/Compiler /t:UpdateXlf ;;
"/run ilverify") pwsh tests/ILVerify/ilverify.ps1 ;;
"/run test-baseline") dotnet test ./FSharp.Compiler.Service.sln --filter "${{ steps.parse.outputs.arguments }}" -c Release || true ;;
*) echo "Unknown command" && exit 1 ;;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review:

This runs user-provided snippet and can pass other arguments to dotnet test, even run other script. This is still in the context of read-only permissions.

esac

- name: Create patch & metadata
id: meta
if: steps.parse.outputs.command
run: |
echo "run_step_outcome=${{ steps.run-cmd.outcome }}" > result
if [[ "${{ steps.run-cmd.outcome }}" == "success" ]]; then
git diff > repo.patch || true
if [ -s repo.patch ]; then echo "hasPatch=true" >> result; else echo "hasPatch=false" >> result; fi
else
echo "hasPatch=false" >> result
fi
cat result

- name: Upload artifacts
uses: actions/upload-artifact@v4
with:
name: cli-results
path: |
repo.patch
result

apply-and-report:
needs: detect-and-run
runs-on: ubuntu-latest
permissions:
contents: write
pull-requests: write
if: needs.detect-and-run.outputs.command != ''
steps:
- name: Extract command to run
uses: actions/github-script@v3
id: command-extractor
with:
result-encoding: string
script: |
if (context.eventName !== "issue_comment") throw "Error: This action only works on issue_comment events.";

// extract the command to run, allowed characters: a-z, A-Z, digits, hyphen, underscore
const regex = /^\/run ([a-zA-Z\d\-\_]+)/;
command = regex.exec(context.payload.comment.body);
if (command == null) throw "Error: No command found in the trigger phrase.";

return command[1];
- name: Get github ref
uses: actions/github-script@v3
id: get-pr
with:
script: |
const result = await github.pulls.get({
pull_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
});
return { "ref": result.data.head.ref, "repository": result.data.head.repo.full_name};
- name: Checkout repo
uses: actions/checkout@v2
with:
repository: ${{ fromJson(steps.get-pr.outputs.result).repository }}
ref: ${{ fromJson(steps.get-pr.outputs.result).ref }}
fetch-depth: 0
- name: Install dotnet
uses: actions/setup-dotnet@v3
with:
global-json-file: global.json
- name: Install dotnet tools
run: dotnet tool restore
- name: Process fantomas command
if: steps.command-extractor.outputs.result == 'fantomas'
id: fantomas
run: dotnet fantomas .
- name: Process xlf command
if: steps.command-extractor.outputs.result == 'xlf'
id: xlf
run: dotnet build src/Compiler /t:UpdateXlf

- name: Commit and push changes
if: steps.fantomas.outcome == 'success' || steps.xlf.outcome == 'success' || steps.ilverify.outcome == 'success'
run: |
# Only commit if there are actual changes
if git diff --quiet; then
echo "No changes to commit, skipping."
exit 0
fi

git config --local user.name "github-actions[bot]"
git config --local user.email "41898282+github-actions[bot]@users.noreply.github.com"
git commit -a -m 'Automated command ran: ${{ steps.command-extractor.outputs.result }}

Co-authored-by: ${{ github.event.comment.user.login }} <${{ github.event.comment.user.id }}+${{ github.event.comment.user.login }}@users.noreply.github.com>'
git push origin HEAD:"refs/heads/$PR_HEAD_REF"\
- name: Post command comment
if: steps.fantomas.outcome == 'success' || steps.xlf.outcome == 'success' || steps.ilverify.outcome == 'success'
uses: actions/github-script@v3
with:
script: |
// Probably, there's more universal way of getting outputs, but my gh-actions-fu is not that good.
var output = ""
if ("${{steps.command-extractor.outputs.result}}" == 'fantomas') {
output = "${{steps.fantomas.outputs.result}}"
} else if ("${{steps.command-extractor.outputs.result}}" == 'xlf') {
output = "${{steps.xlf.outputs.result}}"
} else if ("${{steps.command-extractor.outputs.result}}" == 'ilverify') {
output = "${{steps.ilverify.outputs.result}}"
}
const body = `Ran ${{ steps.command-extractor.outputs.result }}: https://github.com/${context.repo.owner}/${context.repo.repo}/actions/runs/${process.env.GITHUB_RUN_ID}\n${output}`;
await github.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: body
});
- name: Post command failed comment
if: failure()
uses: actions/github-script@v3
with:
script: |
const body = `Failed to run ${{ steps.command-extractor.outputs.result }}: https://github.com/${context.repo.owner}/${context.repo.repo}/actions/runs/${process.env.GITHUB_RUN_ID}`;
await github.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: body
});
- name: Checkout the repository
uses: actions/checkout@v4

- name: Checkout PR branch
run: gh auth setup-git && gh pr checkout ${{ github.event.issue.number }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

- name: Download artifacts
uses: actions/download-artifact@v4
with:
name: cli-results

- name: Read metadata
id: read-meta
run: |
source result
echo "run_step_outcome=$run_step_outcome" >> $GITHUB_OUTPUT
echo "hasPatch=$hasPatch" >> $GITHUB_OUTPUT

- name: Apply and push patch
if: ${{ steps.read-meta.outputs.run_step_outcome == 'success' && steps.read-meta.outputs.hasPatch == 'true' }}
run: |
patch -p1 -s --force < repo.patch || true
git config user.name "GH Actions"
git config user.email "[email protected]"
git add -u
git commit -m "Apply patch from ${{ needs.detect-and-run.outputs.command }}"
upstream=$(git rev-parse --abbrev-ref --symbolic-full-name @{u})
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important: The command must be from a known set, otherwise this is shell injection.
The command is coming from the comment pipeline, only valid command will go trough.

remote=${upstream%%/*}
branch=${upstream#*/}

echo "Pushing to $remote $branch"
git push "$remote" HEAD:"$branch"

- name: Count stats
id: stats
if: ${{ steps.read-meta.outputs.run_step_outcome == 'success' && steps.read-meta.outputs.hasPatch == 'true' }}
run: |
files=$(git diff --name-only HEAD~1 HEAD | wc -l)
lines=$(git diff HEAD~1 HEAD | wc -l)
echo "files=$files" >> $GITHUB_OUTPUT
echo "lines=$lines" >> $GITHUB_OUTPUT
- name: Generate and publish report
if: always()
env:
COMMAND: ${{ needs.detect-and-run.outputs.command }}
OUTCOME: ${{ steps.read-meta.outputs.run_step_outcome }}
PATCH: ${{ steps.read-meta.outputs.hasPatch }}
run: |
# Build the markdown report
report="
# 🔧 CLI Command Report

- **Command:** \`${COMMAND}\`
- **Outcome:** ${OUTCOME}

"

if [[ "$OUTCOME" == "success" ]]; then
if [[ "$PATCH" == "true" ]]; then
report+="✅ Patch applied:
- Files changed: ${{ steps.stats.outputs.files }}
- Lines changed: ${{ steps.stats.outputs.lines }}"
else
report+="✅ Command succeeded, no changes needed."
fi
else
report+="❌ Command **failed** — no patch applied."
fi

# Output to GitHub Actions UI
echo "$report" >> "$GITHUB_STEP_SUMMARY"

# Store for use in next step
echo "$report" > pr_report.md

- name: Comment on PR
if: always()
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
PR_NUMBER: ${{ env.PR_NUMBER }}
run: |
# Use gh CLI to comment with multi-line markdown
gh pr comment ${{ github.event.issue.number }} \
--body-file pr_report.md
20 changes: 20 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,26 @@ If you don't know what a pull request is read this article: <https://help.github
- **DO** submit issues for other features. This facilitates discussion of a feature separately from its implementation, and increases the acceptance rates for pull requests.
- **DO NOT** submit large code formatting changes without discussing with the team first.

#### Repository automation via commands

The following comments in a PR can be used as commands to execute scripts which automate repository maintenance and make it part of the visible diff.
- `/run fantomas` runs `dotnet fantomas .`
- `/run ilverify` updates IL verification baseline
- `/run xlf` refreshes localisation files for translatable strings
- `/run test-baseline ...` runs tests with the `TEST_UPDATE_BSL: 1` environment variable and an argument supplied filter (passed to `dotnet test --filter ..`). Its goal is to refresh baselines.

This code repository uses a lot of baselines - captures for important output - to spot regressions and willingfully accept changes via PR review.
For example, the following errors can appear during CI runs:
- Changes in `Syntax tree tests`
- Differences in generated `IL output`
- Diffrences in produced baseline diagnostics

After identifying a failing test which relies on a baseline, the command can then for example be:
- `/run test-baseline ParseFile` to update parsing tests related to syntactical tree
- `/run test-baseline SurfaceAreaTest` to update the API surface area of FSharp.Compiler.Service
- `/run test-baseline FullyQualifiedName~EmittedIL&FullyQualifiedName~Nullness` to update IL baseline (namespace `EmittedIL`) for tests that touch the `Nullness` feature


### Reviewing pull requests

Our repository gets a high volume of pull requests and reviewing each of them is a significant time commitment. Our team priorities often force us to focus on reviewing a subset of the active pull requests at a given time.
Expand Down
Loading