Skip to content

Commit

Permalink
PyUnitAdapter: Handle an edge case with aborted tests
Browse files Browse the repository at this point in the history
We recently added an exercise to CodeOcean that skipped code execution during the `setUpClass` phase. This resulted in the following output:

```
Ran 0 tests in 0.001s

FAILED (errors=2)
```

As a result, we calculated `count=0`, `failed=2` (and through TestingFrameworkAdapter#augment_output) `passed=-2`. Obviously, this doesn't make sense.

As the output is generated directly through PyUnit, we cannot change it and hence implement a workaround in the adapter to "fake" the number of total tests executed. Not super nice, but working.
  • Loading branch information
MrSerth committed Oct 22, 2024
1 parent f7b0a32 commit 8523133
Showing 1 changed file with 14 additions and 1 deletion.
15 changes: 14 additions & 1 deletion lib/py_unit_adapter.rb
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,19 @@ def parse_output(output)
Sentry.capture_message({stderr: output[:stderr], regex: ASSERTION_ERROR_REGEXP}.to_json)
assertion_error_matches = []
end
{count:, failed: failed + errors, error_messages: assertion_error_matches.flatten.compact_blank}.compact_blank

total_failed = failed + errors

if count < total_failed
# Catch a weird edge case where the test count is less than the failed count.
# This might happen in PyUnit, when a test is failing (by design) during the setUpClass phase.
# In those cases, we might get the following output: Ran 0 tests in 0.001s, FAILED (failures=1)
# Normally, we would calculate the passed tests as count (0) - failed (1) = passed (-1).
# In the given scenario, a negative number of passed tests doesn't make sense.
# Hence, we assume that the count is invalid and increase it by the number of failed tests.
count += total_failed
end

{count:, failed: total_failed, error_messages: assertion_error_matches.flatten.compact_blank}.compact_blank
end
end

0 comments on commit 8523133

Please sign in to comment.