📙 [just] check for Copilot and Claude reviews #25
Conversation
There was a problem hiding this comment.
Pull Request Overview
Adds automation to the PR workflow to watch GitHub Actions checks and then query for Copilot and Claude review comments.
- Introduces a new pr_checks recipe that runs gh pr checks and queries review comments via GraphQL.
- Updates pr recipe to depend on the new pr_checks logic (and bumps internal version comment).
- Removes inline gh pr checks invocation from pr recipe, delegating it to pr_checks.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| # PR create 3.3 | ||
| [group('Process')] | ||
| pr: _has_commits | ||
| pr: _has_commits && pr_checks |
There was a problem hiding this comment.
Using && between dependencies is not valid Just syntax; Just expects a space-separated list of recipe dependencies. This will likely cause a parse error or treat '&&' as an invalid recipe name. Replace with a space: pr: _has_commits pr_checks.
| pr: _has_commits && pr_checks | |
| pr: _has_commits pr_checks |
| gh api graphql \ | ||
| -F owner=$PR_REPO_OWNER -F name=$PR_REPO_NAME -F pr=$PR_NUMBER \ | ||
| --jq '[ .data.repository.pullRequest.reviews.nodes.[] | select(.author.login=="copilot-pull-request-reviewer") | .comments.nodes.[] ]' \ | ||
| -f query=' | ||
| query($name: String!, $owner: String!, $pr: Int!) { | ||
| repository(owner: $owner, name: $name) { | ||
| pullRequest(number: $pr) { | ||
| reviews(last: 20) { | ||
| nodes { | ||
| author { | ||
| resourcePath | ||
| login | ||
| } | ||
| comments(first: 100) { | ||
| nodes { | ||
| body | ||
| path | ||
| originalLine | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| ' | ||
|
|
There was a problem hiding this comment.
Limiting reviews to last: 20 risks missing an earlier Copilot review if more than 20 reviews exist, causing false negatives when checking for Copilot suggestions. Increase the limit or implement pagination (e.g., use after cursors) to ensure all relevant reviews are considered.
| gh api graphql \ | |
| -F owner=$PR_REPO_OWNER -F name=$PR_REPO_NAME -F pr=$PR_NUMBER \ | |
| --jq '[ .data.repository.pullRequest.reviews.nodes.[] | select(.author.login=="copilot-pull-request-reviewer") | .comments.nodes.[] ]' \ | |
| -f query=' | |
| query($name: String!, $owner: String!, $pr: Int!) { | |
| repository(owner: $owner, name: $name) { | |
| pullRequest(number: $pr) { | |
| reviews(last: 20) { | |
| nodes { | |
| author { | |
| resourcePath | |
| login | |
| } | |
| comments(first: 100) { | |
| nodes { | |
| body | |
| path | |
| originalLine | |
| } | |
| } | |
| } | |
| } | |
| } | |
| } | |
| } | |
| ' | |
| # Paginate through all reviews to avoid missing Copilot reviews | |
| COPILOT_COMMENTS=() | |
| REVIEWS_AFTER="" | |
| while : ; do | |
| # Run GraphQL query for a page of reviews | |
| PAGE=$(gh api graphql \ | |
| -F owner=$PR_REPO_OWNER -F name=$PR_REPO_NAME -F pr=$PR_NUMBER -F after="$REVIEWS_AFTER" \ | |
| -f query=' | |
| query($name: String!, $owner: String!, $pr: Int!, $after: String) { | |
| repository(owner: $owner, name: $name) { | |
| pullRequest(number: $pr) { | |
| reviews(first: 100, after: $after) { | |
| nodes { | |
| author { | |
| resourcePath | |
| login | |
| } | |
| comments(first: 100) { | |
| nodes { | |
| body | |
| path | |
| originalLine | |
| } | |
| } | |
| } | |
| pageInfo { | |
| hasNextPage | |
| endCursor | |
| } | |
| } | |
| } | |
| } | |
| } | |
| ') | |
| # Extract Copilot comments from this page | |
| PAGE_COMMENTS=$(echo "$PAGE" | jq '[.data.repository.pullRequest.reviews.nodes[] | select(.author.login=="copilot-pull-request-reviewer") | .comments.nodes[]]') | |
| COPILOT_COMMENTS+=("$PAGE_COMMENTS") | |
| # Check if there are more pages | |
| HAS_NEXT=$(echo "$PAGE" | jq -r '.data.repository.pullRequest.reviews.pageInfo.hasNextPage') | |
| if [[ "$HAS_NEXT" != "true" ]]; then | |
| break | |
| fi | |
| REVIEWS_AFTER=$(echo "$PAGE" | jq -r '.data.repository.pullRequest.reviews.pageInfo.endCursor') | |
| done | |
| # Combine all Copilot comments into one array and output | |
| echo "${COPILOT_COMMENTS[@]}" | jq -s 'add' |
Code Review - PR #25SummaryThis PR adds automated checks for Copilot and Claude reviews after GitHub Actions complete. The changes refactor the pr recipe to delegate the checking workflow to a new pr_checks recipe. ✅ Positive Aspects
🔍 Code Quality Issues1. Inconsistent bash strict mode (.just/gh-process.just:127)The pr_checks recipe uses set -euo pipefail while the pr recipe uses set -euxo pipefail (with -x for tracing). Consider adding -x to pr_checks for consistency and debugging visibility. 2. Missing error handling for jq operations (.just/gh-process.just:133-135)If gh pr view returns unexpected JSON or fields are missing, jq could fail silently or produce empty variables. Consider validating each extraction. 3. Unquoted variables (.just/gh-process.just:133-135)Variables should be quoted to prevent word splitting issues. 4. GraphQL query could fail silently (.just/gh-process.just:137-162)The GraphQL query output is not captured or checked. Consider capturing output and validating it before display. 5. Potential issue with Claude comment check (.just/gh-process.just:166)Using | last could return null if no Claude comments exist. Consider adding // "No Claude review yet" to handle empty case. 🐛 Potential Bugs
🔒 Security Concerns
⚡ Performance Considerations
🧪 Test CoverageMissing: No automated tests for the new pr_checks recipe. Consider manual testing with various scenarios. 📝 Suggestions
📊 Overall AssessmentCode Quality: 7/10 - Good structure but needs robustness improvements ✅ RecommendationAPPROVE with minor changes requested The core functionality is sound and adds value to the workflow. Address the error handling and validation issues before merging, especially around the jq operations and null handling for Claude comments. Review generated by Claude Code |
Code Review for PR #25SummaryThis PR adds automated checking for Copilot and Claude code reviews after GitHub Actions complete. The implementation integrates well with the existing workflow and provides useful developer experience improvements. Positive Aspects ✅
Code Quality Issues1. Error Handling: Missing jq Dependency Check (.just/gh-process.just:135-137)The PR_REPO_NAME=$(echo $PR_META | jq -r ".headRepository.name")Recommendation: Add a dependency check at the beginning of the recipe: if \! command -v jq &> /dev/null; then
echo "Error: jq is required but not installed"
exit 1
fi2. Silent Failure on Empty Results (.just/gh-process.just:168)The Claude comment check uses gh pr view --json comments --jq "[.comments[] | select(.author.login == \"claude\")] | last | .body"When no Claude comments exist, this outputs Recommendation: Add error handling: gh pr view --json comments --jq "[.comments[] | select(.author.login == \"claude\")] | last | .body // \"No Claude review found\""3. Incomplete Copilot Output Handling (.just/gh-process.just:139-164)The Copilot GraphQL query extracts review comments but does not format or display them in a user-friendly way. The output is just raw JSON to stdout. Recommendation: Add formatting similar to the Claude output to make the results more readable. 4. Inconsistent Error Handling Between Copilot and Claude ChecksThe Copilot check uses GraphQL API while Claude uses REST API, but neither has explicit error handling for API failures. Recommendation: Wrap both sections in error checking to gracefully handle API failures. Potential Issues5. Hardcoded Author Login (.just/gh-process.just:168)The Claude check uses
Recommendation: Consider making this configurable or add a comment explaining the assumption. 6. Race Condition Risk (.just/gh-process.just:131)The recipe runs Recommendation: Consider adding a small sleep or retry logic: gh pr checks --watch -i 5
sleep 2 # Give review bots time to post7. CRITICAL: Missing Recipe Dependency (.just/gh-process.just:19)The Critical Bug: Recommendation: Move Performance Considerations8. GraphQL Query Efficiency (.just/gh-process.just:139-164)The query fetches Recommendation: Consider limiting to more reasonable defaults (e.g., Security Concerns9. Temp File Handling (.just/gh-process.just:28, 44)Good practice: The PR properly cleans up the temp file at line 44. No security issues here. ✅ 10. Command Injection Risk - Low Priority (.just/gh-process.just:135-137)Using PR_REPO_NAME=$(jq -r ".headRepository.name" <<< "$PR_META")Test CoverageNo tests are included for the new Recommendation: Consider adding a comment documenting expected behavior for future maintainers. Minor Issues
RecommendationRequest Changes - The critical bug in item #7 (circular dependency/timing issue) must be fixed before merging. The Other issues are lower priority but should be addressed for production readiness. Great work on improving the developer experience! The approach is solid, just needs the dependency/timing fix. 🚀 🤖 Generated with Claude Code |
Context 🌍
I've been annoyed with missing Copilot reviews while working in a terminal-focused development workflow. I asked in the github cli repo about this problem and I was amazed to get an awesome answer with exactly what I needed
Done ✅
Created a new subcommand
just pr_cheskwhich watches for the Github Actions to finish running and then it checks for Copilot code reviews and then it checks for Claude code reviews.Trust but Verify ™️
I developed this in one repo, verified and refined it in a private repo and so this PR is the third one that has used this code happily. I've also run the new subcommand separately quite a few times.
Meta 🖥️
It is time for a new meta...
(Automated in
just/gh-process.just.)