Skip to content

Add common command CIs #3449

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 4 commits into
base: master
Choose a base branch
from
Draft

Add common command CIs #3449

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
81065a8
add new CI scripts and configs
Lmh-java Mar 15, 2025
06df1b7
revise timeout and commands
Lmh-java Mar 15, 2025
05c6cd9
print log upon error
Lmh-java Mar 15, 2025
7eded6d
add thunderscope command and proto replay
Lmh-java Mar 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions .github/workflows/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,3 +177,33 @@ jobs:
name: yellow-ai-vs-ai-proto-logs
path: |
/tmp/tbots/yellow/proto_*

common-commands:
strategy:
# TODO: set fail-fast to true in production.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remember to remove TODO before merging

fail-fast: false
matrix:
commands:
- ../scripts/safe_run.sh bazel run --run_under="xvfb-run" //software/thunderscope:thunderscope_main -- --run_blue --run_diagnostics --interface lo --keyboard_estop --ci_mode
- ../scripts/safe_run.sh bazel run --run_under="xvfb-run" //software/ai/hl/stp/tactic/goalie:goalie_tactic_test -- --enable_thunderscope
- ../scripts/safe_run.sh bazel run --run_under="xvfb-run" //software/thunderscope:thunderscope_main -- --enable_autoref --ci_mode && ../scripts/safe_run.sh bazel run --run_under="xvfb-run" //software/thunderscope:thunderscope_main -- blue_log="$(find /tmp/tbots/blue -maxdepth 1 -type d -name 'proto_*' -printf '/tmp/tbots/blue/%f\n' 2>/dev/null | head -n1)"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does the second half of this command do?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the delay, I am finally done work.

../scripts/safe_run.sh bazel run --run_under="xvfb-run" //software/thunderscope:thunderscope_main -- --enable_autoref --ci_mode This first half runs thunderscope once

This part tries to find the previous replay log and display the logs:
../scripts/safe_run.sh bazel run --run_under="xvfb-run" //software/thunderscope:thunderscope_main -- blue_log="$(find /tmp/tbots/blue -maxde4pth 1 -type d -name 'proto_*' -printf '/tmp/tbots/blue/%f\n' 2>/dev/null | head -n1)

This is not ideal, I am trying to think of a way to handle this. Maybe I should allow a new argument for our thunderscope, so we can specify the location for the replay log to save, so that we don't need to lookup for it.


name: Sanity Check on Common Commands
runs-on: ubuntu-20.04
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can use the ubuntu-22.04 runner

steps:
- uses: actions/checkout@v4

- name: Environment Setup
run: |
"${GITHUB_WORKSPACE}"/environment_setup/setup_software.sh

- name: Build Thunderscope
run: |
cd src
bazel build //software/thunderscope:thunderscope_main

- name: Check Common Commands
run: |
cd src
${{ matrix.commands }}

52 changes: 52 additions & 0 deletions scripts/safe_run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
#!/bin/bash
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

safe_run.sh is sort of a weird name that doesn't immediately tell you what this file does

But tbh I don't know a better name... maybe ci_runner.sh is better. At least it tells you that it's only relevant for CI.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a high-level description of this script at the top of this file.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you re-write this script with python, you can use subprocess and get stdout and stderr logs. Then, you can print the error messages. Because right now, there isn't any feedback for the developers and it will be confusing when this CI job suddenly fails.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, sounds good!
I was trying to use 2> and > to redirect stdout and stderr, but it didn't seem to work.


# Timeout in seconds
# When the time is up and no error was shown, this test will pass
TIME_LIMIT=120 # 2 minutes

# Match Python traceback
ERROR_PATTERN="Traceback (most recent call last):"

# Temporary log file
LOG_FILE=$(mktemp)

# Run the command and record the log
"$@" &> "$LOG_FILE" &
CMD_PID=$!

echo "Process Running in Wrapper with Timeout $TIME_LIMIT ..."

# Time the process
SECONDS=0
while kill -0 $CMD_PID 2>/dev/null; do
# Check if time is up
if [ $SECONDS -ge $TIME_LIMIT ]; then
echo "Time limit reached, stopping process: $CMD_PID"
kill -SIGINT $CMD_PID
wait $CMD_PID
exit 0 # Upon time out and no error, returns 0 status code
fi

# Check if the log contains Traceback
if grep -q "$ERROR_PATTERN" "$LOG_FILE"; then
echo "[Error detected] Potential error found in command output!"
kill -SIGINT $CMD_PID
wait $CMD_PID
exit 1
fi

sleep 1 # Run this loop once per second
done

cat $LOG_FILE
# Get the exit code of the process
wait $CMD_PID
EXIT_CODE=$?

# Clean up log file
rm -f "$LOG_FILE"

# Exit with the command status code
exit $EXIT_CODE


Loading