Skip to content

Pytest validation #3123

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 81 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
d2140c1
crease defense and offense updates
PhilD71 Jun 3, 2022
c101d0a
more test updates
PhilD71 Jun 3, 2022
1efe449
more stuff
PhilD71 Jun 3, 2022
679f5b5
more tests
PhilD71 Jun 3, 2022
bddfc6d
play updates
PhilD71 Jun 9, 2022
6cd7863
Merge branch 'master' of github.com:UBC-Thunderbots/Software into phi…
PhilD71 Jun 10, 2022
84ad9e4
test_updates
PhilD71 Jun 22, 2022
caf5522
Merge branch 'master' into phil/pytest_one
PhilD71 Jun 22, 2022
2aea34d
minor cleanup
PhilD71 Jun 22, 2022
3b3f270
kickoff_play test concept
PhilD71 Jun 23, 2022
e7f15c3
Merge branch 'master' into phil/pytest_one
PhilD71 Jun 24, 2022
6b2b5ad
updated free kick play test
PhilD71 Jun 30, 2022
4a6d5b8
even more free_kick_play updates
PhilD71 Jun 30, 2022
caf6175
added penalty_kick_play_test
PhilD71 Jul 7, 2022
0ee4e2f
validated penalty kick
PhilD71 Jul 7, 2022
0f2d160
Merged with master
PhilD71 Sep 7, 2022
9e0ecea
removed comment
PhilD71 Oct 8, 2022
bb41198
Merge branch 'phil/pytest_one' of github.com:PhilD71/Software into ph…
PhilD71 Oct 13, 2022
634f4ff
removed validation
PhilD71 Oct 13, 2022
da16d0e
formatting and small test exclusions
PhilD71 Oct 13, 2022
0f8fc26
disabled penalty kick play
PhilD71 Oct 13, 2022
b34dcf4
Updated GPIO to follow codebase naming guidelines. Ex: instances of "…
suchirss Oct 14, 2022
f4c09df
Merge branch 'master' into phil/pytest_one
PhilD71 Jan 19, 2023
7791e35
fixed up comments, still some to go
PhilD71 Jan 20, 2023
b3b813b
Just added a comment on line 79 that clarifies my task.
suchirss Jan 27, 2023
be83cd7
Added always-validation for 6 robots in friendly half.
suchirss Feb 19, 2023
0594740
Removed unnecessary comments.
suchirss Feb 19, 2023
24dfaf2
test commit
suchirss Jun 16, 2023
bffbe18
added eventually validation for ball exiting center circle
suchirss Jun 16, 2023
2526709
Decided on new validation strategy where a certain amount of robots d…
suchirss Jun 19, 2023
1eac5a6
changed centreCircle() to centerCircle() (spelling issue)
suchirss Jun 19, 2023
1ea49a6
Ran formatting script
suchirss Jun 20, 2023
b20c9e9
Implemented pytest parametrize feature in always validation set. Adde…
suchirss Jun 26, 2023
43b87b7
Added circular region of 0.05 radius with focus at (0,0) to BallEvent…
suchirss Jun 26, 2023
3758960
Revert "added eventually validation for ball exiting center circle"
suchirss Jun 26, 2023
2583f58
pulled from upstream
suchirss Jun 26, 2023
a7be834
Merge branch 'master' into pytest-validation
suchirss Jun 26, 2023
3927b21
Ran formatting script
suchirss Jun 26, 2023
aae3ba8
Changed attributes to have correct capitalization (i.e: tbots.Field i…
suchirss Jun 26, 2023
29b8ebc
Fixed some more capitalization
suchirss Jun 26, 2023
6b8c492
Started creating orValidation.py extension which allows multiple vali…
suchirss Oct 15, 2023
6c3ea41
Futher developed get_validation_status, get_validation_type, and get_…
suchirss Nov 11, 2023
b366b5c
Further developed implementation of or validation within kickoff vali…
suchirss Nov 11, 2023
4f5e18f
Added validation of 6 robots within friendlyHalf + centreCircle in ad…
suchirss Nov 18, 2023
2538d9b
Resolves some errors when running kickoff_play_test.py
suchirss Nov 19, 2023
eb67efa
Modified NumberOfRobotsEnterRegion() in robot_enters_region.py to acc…
suchirss Jan 20, 2024
6e4ab7d
Fixed geometry concatenation in get_validation_geometry in or_validat…
suchirss Jan 20, 2024
c7a7d2d
Changed order of gamecontroller commands so NORMAL_START happens afte…
suchirss Jan 21, 2024
6a149fe
Disabled case of kickoff_play_test.py where is_friendly_test is False…
suchirss Jan 21, 2024
22121c6
Merge branch 'master' of github.com:UBC-Thunderbots/Software into pyt…
suchirss Jan 27, 2024
1de63b0
Typo: changed BLU to BLUE.
suchirss Jan 27, 2024
98687d6
Merge branch 'master' of github.com:UBC-Thunderbots/Software into pyt…
suchirss Jan 27, 2024
43906bc
PR changes: fixed some naming, deleted duplicate file.
suchirss Mar 30, 2024
af77463
Fixed GPIO naming. Gpio -> GPIO in comments.
suchirss Mar 30, 2024
bc891f7
Deleted robot_halt.py and associated build string.
suchirss Mar 30, 2024
87d2827
Took code segment that checks all robots are in friendly half + cente…
suchirss Mar 30, 2024
deb4d21
More naming changes. Tbots -> tbots_cpp.
suchirss Mar 30, 2024
f3930bf
Made RobotEntersRegion a child class of NumberOfRobotEntersRegion
suchirss Mar 30, 2024
590719f
Renamed NumberOfRobotsEntersRegion to MinNumberOfRobotsEntersRegion. …
suchirss Mar 30, 2024
d405467
Merge branch 'master' of github.com:UBC-Thunderbots/Software into pyt…
suchirss Mar 31, 2024
f7fa44a
Ran formatting script
suchirss Mar 31, 2024
91c000e
[pre-commit.ci lite] apply automatic fixes
pre-commit-ci-lite[bot] Mar 31, 2024
0966c4d
Fixed GPIO casing issue.
suchirss May 8, 2024
1ff45af
Removed unused imports. Ran formatting script.
suchirss May 8, 2024
06080b3
Added back import in or_validation.py.
suchirss May 8, 2024
16c411e
Merge branch 'master' of github.com:UBC-Thunderbots/Software into pyt…
suchirss May 8, 2024
da100d5
Merge remote-tracking branch 'origin/pytest-validation' into pytest-v…
suchirss May 8, 2024
bfb8626
Moved TypeError check to initialization function in or_validation.py.…
suchirss May 12, 2024
ee7846c
Merge branch 'master' of github.com:UBC-Thunderbots/Software into pyt…
suchirss May 12, 2024
a3a4344
Fixed or_validation.py such that kickoff_play_test.py passes when tes…
suchirss May 12, 2024
e38f098
Merge branch 'master' of github.com:UBC-Thunderbots/Software into pyt…
suchirss Jan 11, 2025
1cb1eeb
[pre-commit.ci lite] apply automatic fixes
pre-commit-ci-lite[bot] Jan 11, 2025
12d5d0e
Removed check that all validation class inputs must be of the same ty…
suchirss Jan 11, 2025
0a18f61
Removed check that all validation class inputs must be of the same ty…
suchirss Jan 11, 2025
91564d1
Merge remote-tracking branch 'origin/pytest-validation' into pytest-v…
suchirss Jan 11, 2025
c660224
Merge branch 'master' of github.com:UBC-Thunderbots/Software into pyt…
suchirss Jan 20, 2025
f2d4024
Modified kickoff_play_test so the validation that checks that all rob…
suchirss Feb 17, 2025
8050615
Kickoff validation areas and friendly robot behaviour corrected.
suchirss Mar 1, 2025
82a89c1
Added back validation type consistency check to OrValidation class.
suchirss Mar 1, 2025
810d8f8
Merge branch 'master' of github.com:UBC-Thunderbots/Software into pyt…
suchirss Mar 1, 2025
468fa4d
Ran formatting script
suchirss Mar 1, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
212 changes: 148 additions & 64 deletions src/software/ai/hl/stp/play/kickoff_play_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,89 +4,173 @@

import software.python_bindings as tbots_cpp
from proto.play_pb2 import Play, PlayName
from software.simulated_tests.robot_enters_region import *
from software.simulated_tests.ball_enters_region import *
from software.simulated_tests.ball_moves_from_rest import *
from proto.import_all_protos import *
from proto.message_translation.tbots_protobuf import create_world_state
from proto.ssl_gc_common_pb2 import Team
from software.simulated_tests.or_validation import OrValidation


@pytest.mark.parametrize("is_friendly_test", [True, False])
def test_kickoff_play(simulated_test_runner, is_friendly_test):
def setup(*args):
# starting point must be Point
ball_initial_pos = tbots_cpp.Point(0, 0)

# Setup Bots
blue_bots = [
tbots_cpp.Point(-3, 2.5),
tbots_cpp.Point(-3, 1.5),
tbots_cpp.Point(-3, 0.5),
tbots_cpp.Point(-3, -0.5),
tbots_cpp.Point(-3, -1.5),
tbots_cpp.Point(-3, -2.5),
]

yellow_bots = [
tbots_cpp.Point(1, 0),
tbots_cpp.Point(1, 2.5),
tbots_cpp.Point(1, -2.5),
tbots_cpp.Field.createSSLDivisionBField().enemyGoalCenter(),
tbots_cpp.Field.createSSLDivisionBField()
.enemyDefenseArea()
.negXNegYCorner(),
tbots_cpp.Field.createSSLDivisionBField()
.enemyDefenseArea()
.negXPosYCorner(),
]

blue_play = Play()
yellow_play = Play()

# Game Controller Setup
ball_initial_pos = tbots_cpp.Point(0, 0)

# Setup Bots
blue_bots = [
tbots_cpp.Point(-3, 2.5),
tbots_cpp.Point(-3, 1.5),
tbots_cpp.Point(-3, 0.5),
tbots_cpp.Point(-3, -0.5),
tbots_cpp.Point(-3, -1.5),
tbots_cpp.Point(-3, -2.5),
]

yellow_bots = [
tbots_cpp.Point(1, 0),
tbots_cpp.Point(1, 2.5),
tbots_cpp.Point(1, -2.5),
tbots_cpp.Field.createSSLDivisionBField().enemyGoalCenter(),
tbots_cpp.Field.createSSLDivisionBField().enemyDefenseArea().negXNegYCorner(),
tbots_cpp.Field.createSSLDivisionBField().enemyDefenseArea().negXPosYCorner(),
]

blue_play = Play()
yellow_play = Play()

# Game Controller Setup
simulated_test_runner.gamecontroller.send_gc_command(
gc_command=Command.Type.STOP, team=Team.UNKNOWN
)

if is_friendly_test:
simulated_test_runner.gamecontroller.send_gc_command(
gc_command=Command.Type.STOP, team=Team.UNKNOWN
gc_command=Command.Type.KICKOFF, team=Team.BLUE
)
blue_play.name = PlayName.KickoffFriendlyPlay
yellow_play.name = PlayName.KickoffEnemyPlay
else:
simulated_test_runner.gamecontroller.send_gc_command(
gc_command=Command.Type.NORMAL_START, team=Team.BLUE
gc_command=Command.Type.KICKOFF, team=Team.YELLOW
)
if is_friendly_test:
simulated_test_runner.gamecontroller.send_gc_command(
gc_command=Command.Type.KICKOFF, team=Team.BLUE
blue_play.name = PlayName.KickoffEnemyPlay
yellow_play.name = PlayName.KickoffFriendlyPlay

simulated_test_runner.gamecontroller.send_gc_command(
gc_command=Command.Type.NORMAL_START, team=Team.BLUE
)

# Force play override here
simulated_test_runner.blue_full_system_proto_unix_io.send_proto(Play, blue_play)
simulated_test_runner.yellow_full_system_proto_unix_io.send_proto(Play, yellow_play)

# Create world state
simulated_test_runner.simulator_proto_unix_io.send_proto(
WorldState,
create_world_state(
yellow_robot_locations=yellow_bots,
blue_robot_locations=blue_bots,
ball_location=ball_initial_pos,
ball_velocity=tbots_cpp.Vector(0, 0),
),
)

# Always Validation
always_validation_sequence_set = [[]]
if is_friendly_test:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this if statement could probably be condensed a bit, so this can be common between both branches:

always_validation_sequence_set[0].append(
            NumberOfRobotsAlwaysStaysInRegion(
                regions=[
                    tbots.Field.createSSLDivisionBField().friendlyHalf(),
                    tbots.Field.createSSLDivisionBField().centerCircle(),
                ],
                req_robot_cnt=6,
            )
        )
always_validation_sequence_set[0].append(
            OrValidation(
                [
                    NumberOfRobotsAlwaysStaysInRegion(
                        regions=[tbots.Field.createSSLDivisionBField().centerCircle()],
                        req_robot_cnt=0,
                    ),
                ]
            )

and you can just append this to the OR validation in the friendly case

 NumberOfRobotsAlwaysStaysInRegion(
           regions=[tbots.Field.createSSLDivisionBField().centerCircle()],
           req_robot_cnt=1,
),

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

# Checks that either 0 or 1 robots are in centerCircle OR ball moves from center point
always_validation_sequence_set[0].append(
OrValidation(
[
BallAlwaysMovesFromRest(
position=tbots_cpp.Point(0, 0), threshold=0.01
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

),
NumberOfRobotsAlwaysStaysInRegion(
regions=[
tbots_cpp.Field.createSSLDivisionBField().centerCircle()
],
req_robot_cnt=0,
),
NumberOfRobotsAlwaysStaysInRegion(
regions=[
tbots_cpp.Field.createSSLDivisionBField().centerCircle()
],
req_robot_cnt=1,
),
]
)
blue_play.name = PlayName.KickoffFriendlyPlay
yellow_play.name = PlayName.KickoffEnemyPlay
else:
simulated_test_runner.gamecontroller.send_gc_command(
gc_command=Command.Type.KICKOFF, team=Team.YELLOW
)

# Checks that there are 6 friendly robots in friendlyHalf + centerCircle + friendlyGoal OR ball moves from center point
always_validation_sequence_set[0].append(
OrValidation(
[
BallAlwaysMovesFromRest(
position=tbots_cpp.Point(0, 0), threshold=0.01
),
NumberOfRobotsAlwaysStaysInRegion(
regions=[
tbots_cpp.Field.createSSLDivisionBField().friendlyHalf(),
tbots_cpp.Field.createSSLDivisionBField().friendlyGoal(),
tbots_cpp.Field.createSSLDivisionBField().centerCircle(),
],
req_robot_cnt=6,
),
]
)
)

else:
# Checks that 0 robots are in centerCircle OR ball moves from center point
always_validation_sequence_set[0].append(
OrValidation(
[
BallAlwaysMovesFromRest(
position=tbots_cpp.Point(0, 0), threshold=0.01
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

),
NumberOfRobotsAlwaysStaysInRegion(
regions=[
tbots_cpp.Field.createSSLDivisionBField().centerCircle()
],
req_robot_cnt=0,
),
]
)
blue_play.name = PlayName.KickoffEnemyPlay
yellow_play.name = PlayName.KickoffFriendlyPlay
)

# Force play override here
simulated_test_runner.blue_full_system_proto_unix_io.send_proto(Play, blue_play)
simulated_test_runner.yellow_full_system_proto_unix_io.send_proto(
Play, yellow_play
# Checks that there are 6 enemy robots in friendlyHalf + centerCircle + friendlyGoal OR ball moves from center point
always_validation_sequence_set[0].append(
OrValidation(
[
BallAlwaysMovesFromRest(
position=tbots_cpp.Point(0, 0), threshold=0.01
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

),
NumberOfRobotsAlwaysStaysInRegion(
regions=[
tbots_cpp.Field.createSSLDivisionBField().friendlyHalf(),
tbots_cpp.Field.createSSLDivisionBField().friendlyGoal(),
],
req_robot_cnt=6,
),
]
)
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you could reduce the duplication in this code a bit:

ball_moves_at_rest_validation = BallAlwaysMovesFromRest(
    position=tbots_cpp.Point(0, 0),
    threshold=0.01
)

expected_centre_circle_or_validations_set = [
    ball_moves_at_rest_validation,
    NumberOfRobotAlwaysStaysInRegion(
        regions = [
            tbots_cpp.Field.createSSLDivisionBField().centerCircle()
        ],
        req_robot_cnt = 0
    )
]

friendly_regions = [tbots_cpp.Field.createSSLDivisionBField().friendlyHalf()]

if friendly_test:
    expected_centre_circle_or_validations_set.append(
        NumberOfRobotsAlwaysStaysInRegion(
            regions=[
                tbots_cpp.Field.createSSLDivisionBField().centerCircle()
            ],
            req_robot_cnt = 1
        )
    )
else:
   friendly_regions.append(tbots_cpp.Field.createSSLDivisionBField().centerCircle())

expected_robot_regions_or_validations_set = [
    ball_moves_at_rest_validation,
    NumberOfRobotAlwaysStaysInRegion(
       regions=friendly_regions,
       req_robot_cnt=6
    )
]

always_validation_sequence_set[0] = [
    OrValidation(expected_centre_circle_or_validations_set)),
    OrValidation(expected_robot_regions_or_validations_set)
]


# Create world state
simulated_test_runner.simulator_proto_unix_io.send_proto(
WorldState,
create_world_state(
yellow_robot_locations=yellow_bots,
blue_robot_locations=blue_bots,
ball_location=ball_initial_pos,
ball_velocity=tbots_cpp.Vector(0, 0),
),
eventually_validation_sequence_set = [[]]

# Eventually Validation
if is_friendly_test:
# Checks that ball leaves center point by 0.05 meters within 10 seconds of kickoff
eventually_validation_sequence_set[0].append(
BallEventuallyExitsRegion(
regions=[tbots_cpp.Circle(ball_initial_pos, 0.05)]
)
)

# TODO- #2809 Validation
# params just have to be a list of length 1 to ensure the test runs at least once
simulated_test_runner.run_test(
setup=setup,
params=[0],
inv_always_validation_sequence_set=[[]],
inv_eventually_validation_sequence_set=[[]],
ag_always_validation_sequence_set=[[]],
ag_eventually_validation_sequence_set=[[]],
inv_eventually_validation_sequence_set=eventually_validation_sequence_set,
inv_always_validation_sequence_set=always_validation_sequence_set,
test_timeout_s=10,
)

Expand Down
2 changes: 1 addition & 1 deletion src/software/simulated_tests/ball_enters_region.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ def get_validation_status(self, world) -> ValidationStatus:
def get_validation_geometry(self, world) -> ValidationGeometry:
"""Returns the underlying geometry this validation is checking

:param world: The world msg to create v alidation geometry from
:param world: The world msg to create validation geometry from
:return: ValidationGeometry containing geometry to visualize

"""
Expand Down
16 changes: 8 additions & 8 deletions src/software/simulated_tests/or_validation.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,13 @@

class OrValidation(Validation):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really neat! 💯

def __init__(self, validations):
"""An or extension to the validation function"""
"""An OR extension to the validation function"""
assert len(validations) > 0
validation_type_initial = validations[0].get_validation_type()
for validation in validations:
validation_type = validation.get_validation_type()
if validation_type != validation_type_initial:
raise TypeError("Type of validation instances is not consistent")
self.validations = validations

def get_validation_status(self, world):
Expand All @@ -32,10 +38,4 @@ def get_validation_geometry(self, world):
return validation_geometry

def get_validation_type(self, world):
validation_type_initial = self.validations[0].get_validation_type

for validation in self.validations:
validation_type = validation.get_validation_type
if validation_type != validation_type_initial:
raise TypeError("type of validation instances is not consistent")
return validation_type_initial
return ValidationType
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
return ValidationType
return self.validations[0].get_validation_type(world)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bump

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the constructor should check that the validation types are consistent

def __init__(self, validations):
        """An OR extension to the validation function"""
        assert len(validations) > 0
        validation_type_initial = validations[0].get_validation_type()
        for validation in validations:
            validation_type = validation.get_validation_type()
            if validation_type != validation_type_initial:
                raise TypeError("Type of validation instances is not consistent")
        self.validations = validations

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do validation types put into the function need to be consistent?

If we wanted to check if either an always validation or eventually validation are passing, then the OrValidation class would not be able to support this (provided that this check is kept).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our underlying validation code doesn't support mixing eventually and always validations:

def run_validation_sequence_sets(
. So, if we try to do something like that, we should warn the programmer that it probably won't work

Loading
Loading