[Bug] Failed cleanup mark test as passed #2531
Description
chainsaw version Version
v0.2.13
Description
With v0.2.13, when a step cleanup is ERROR, the report mark the test as "Passed" and the return code is 0 instead of "failed" (and return 1).
With v0.2.12, the report is "Failed" as expected.
Steps to reproduce
I deploy an Ingress with a finalizer that chainsaw can't delete :
- name: deploy-ingress
try:
- apply:
file: samples/ingress.yaml
- assert:
file: samples/ingress.yamlIn the output I have that:
| 01:34:41 | my-test | deploy-ingress | CLEANUP | ERROR |
=== ERROR
Get "https://10.308.194.15/apis/networking.k8s.io/v1/namespaces/test-namespace/ingresses/ingress-sample": context deadline exceeded
| 01:34:41 | my-test | deploy-ingress | CLEANUP | END |
...
--- FAIL: chainsaw (5.21s)
--- FAIL: chainsaw/my-test (8.29s)
FAIL
Tests Summary...
- Passed tests 1
- Failed tests 0
- Skipped tests 0
Done.
Expected behavior
I did the exact same test with v0.2.12 and this is the output:
| 01:31:24 | my-test | deploy-ingress | CLEANUP | ERROR |
=== ERROR
Get "https://10.308.194.15/apis/networking.k8s.io/v1/namespaces/test-namespace/ingresses/ingress-sample": context deadline exceeded
| 01:31:24 | my-test | deploy-ingress | CLEANUP | END |
...
--- FAIL: chainsaw (5.18s)
--- FAIL: chainsaw/my-test (8.82s)
FAIL
Tests Summary...
- Passed tests 0
- Failed tests 1
- Skipped tests 0
Done with failures.
Error: some tests failed
Screenshots
No response
Logs
Slack discussion
No response
Troubleshooting
- I have searched other issues in this repository and mine is not recorded.