Test run summary - tests results

[nirs]

When we finish to run the tests we want to have a machine and human readable summary of the test run. The summary should be written to the report directory.

Initial draft of the summary file:

tests:
- workload: deploy
  deployer: appset
  pvcspec: rbd
  status: pass
- workload: deploy
  deployer: appset
  pvcspec: cephfs
  status: fail

Issues

How to describe setup and cleanup

Test run contains:

1. Setup environment
2. Run tests

Test clean contains

1. Run clean
2. Clean environment

The report should describe this flow, including errors in test or in the setup/cleanup steps.

Test run:

steps:
- name: setup
  status: passed
- name: tests
  tests:
  - workload: deploy
    deployer: appset
    pvcspec: rbd
    status: passed
  - workload: deploy
    deployer: appset
    pvcspec: cephfs
    status: passed

Test clean:

steps:
- name: tests
  tests:
  - workload: deploy
    deployer: appset
    pvcspec: rbd
    status: passed
  - workload: deploy
    deployer: appset
    pvcspec: cephfs
    status: passed
- name: cleanup
  status: passed

How to describe failures

Need to think how to describe the failure with enough details.

In go test we have this output:

--- PASS: TestDR (7.14s)
    --- PASS: TestDR/appset-deploy-cephfs-busybox (670.70s)
        --- PASS: TestDR/appset-deploy-cephfs-busybox/Deploy (5.40s)
        --- PASS: TestDR/appset-deploy-cephfs-busybox/Enable (126.50s)
        --- PASS: TestDR/appset-deploy-cephfs-busybox/Failover (115.80s)
        --- PASS: TestDR/appset-deploy-cephfs-busybox/Relocate (367.66s)
        --- PASS: TestDR/appset-deploy-cephfs-busybox/Disable (55.21s)
        --- PASS: TestDR/appset-deploy-cephfs-busybox/Undeploy (0.13s)

It is nice but contains lot of redundant info and noise, and not machine readable.

Since our tests are serial (deploy, protect, failover, ...) we don't need to show all the steps to describe the failure. The name of the failed step is enough. However the timing info is useful. Maybe we can do this:

tests:
- workload: deploy
  deployer: appset
  pvcspec: cephfs
  status: fail
  time: 670.70s
  steps:
  - name: deploy
    time: 5.40s
  - name: protect
    time: 126.50s
  - name: failover
    time: 115.80s
  - name: relocate
    time: 423.00s