Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: Add some pytest marks and reporting. #4401

Closed
wants to merge 1 commit into from

Conversation

dougch
Copy link
Contributor

@dougch dougch commented Feb 1, 2024

Resolved issues:

none

Description of changes:

While sorting out the integration tests needing attention when running in a nix environment, we were looking for a way to track what's left. Using pytest's mark feature, simply as a reporting mechanism is helpful. This PR is intended to spark discussion about the use of marks.

Sample output

>nix develop
>pytestmark_report nix
+---------------------------------------------------------+
| Tests with the nix mark                                 |
+---------------------------------------------------------+
tests/integrationv2/test_buffered_send.py: 7651
tests/integrationv2/test_client_authentication.py: 1308
tests/integrationv2/test_early_data.py: 3964
tests/integrationv2/test_fragmentation.py: 72
tests/integrationv2/test_hello_retry_requests.py: 457
tests/integrationv2/test_key_update.py: 7
tests/integrationv2/test_npn.py: 1010
tests/integrationv2/test_ocsp.py: 576
tests/integrationv2/test_pq_handshake.py: 82
tests/integrationv2/test_record_padding.py: 126
tests/integrationv2/test_renegotiate.py: 1010
tests/integrationv2/test_session_resumption.py: 7412
tests/integrationv2/test_signature_algorithms.py: 6126
tests/integrationv2/test_sni_match.py: 88
tests/integrationv2/test_version_negotiation.py: 6673
tests/integrationv2/test_well_known_endpoints.py: 480

+---------------------------------------------------------+
| Tests without the nix mark                              |
+---------------------------------------------------------+
tests/integrationv2/test_cross_compatibility.py: 3336
tests/integrationv2/test_dynamic_record_sizes.py: 1278
tests/integrationv2/test_external_psk.py: 4159
tests/integrationv2/test_happy_path.py: 8552
tests/integrationv2/test_pq_handshake.py: 5
tests/integrationv2/test_record_padding.py: 127
tests/integrationv2/test_renegotiate_apache.py: 28
tests/integrationv2/test_signature_algorithms.py: 6126

Call-outs:

This changes nothing about how tests are currently run.

Testing:

How is this change tested (unit tests, fuzz tests, etc.)? locally

Is this a refactor change? If so, how have you proved that the intended behavior hasn't changed?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@github-actions github-actions bot added the s2n-core team label Feb 1, 2024
@dougch dougch requested a review from jmayclin February 8, 2024 00:23
@dougch dougch closed this Feb 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant