Add test report #48

nirs · 2025-03-19T21:39:08Z

Add test report for test-run and test-clean commands. When a test run or
clean is completed, we write a yaml report with the name of the command.
The report include information about the host, ramenctl, and the test
command.

Example test run command

% ramenctl test run -o test
⭐ Using report "test"
⭐ Using config "config.yaml"
✅ Environment setup                                                          
✅ Application "appset-deploy-rbd" deployed                                   
✅ Application "appset-deploy-rbd" protected                                  
✅ Application "appset-deploy-rbd" failed over                                
✅ Application "appset-deploy-rbd" relocated                                  
✅ Application "appset-deploy-rbd" unprotected                                
✅ Application "appset-deploy-rbd" undeployed                                 
✅ passed (1 passed, 0 failed, 0 skipped)

Example test clean command

% ramenctl test clean -o test
⭐ Using report "test"
⭐ Using config "config.yaml"
✅ Application "appset-deploy-rbd" unprotected                                
✅ Application "appset-deploy-rbd" undeployed                                 
✅ Environment cleaned                                                        
✅ passed (1 passed, 0 failed, 0 skipped)

Example report

Report contents

% tree test
test
├── test-clean.log
├── test-clean.yaml
├── test-run.log
└── test-run.yaml

test-run.yaml

host:
  arch: arm64
  cpus: 12
  os: darwin
name: test-run
ramenctl:
  commit: b7f254c8a8310df45e119b76ac83e72e7a90a55e
  version: v0.1.0-13-gb7f254c
status: passed
steps:
- name: setup
  status: passed
- name: tests
  status: passed
  tests:
  - deployer: appset
    pvcSpec: rbd
    status: passed
    workload: deploy
summary:
  failed: 0
  passed: 1
  skipped: 0

test-clean.yaml

host:
  arch: arm64
  cpus: 12
  os: darwin
name: test-clean
ramenctl:
  commit: b7f254c8a8310df45e119b76ac83e72e7a90a55e
  version: v0.1.0-13-gb7f254c
status: passed
steps:
- name: tests
  status: passed
  tests:
  - deployer: appset
    pvcSpec: rbd
    status: passed
    workload: deploy
- name: cleanup
  status: passed
summary:
  failed: 0
  passed: 1
  skipped: 0

Report tarball

test.tar.gz

Fixes #34
Fixes #40

parikshithb

Tried this out locally, Looks great !!!

nirs · 2025-03-21T16:36:58Z

Changes in latest version:

Add Command.Setup() and Command.Cleanup()
Add Command.RunTest() and Command.CleanTest()
Keep the report in the Command
Update the report in Setup(), Cleanup(), RunTest(), CleanTest()
Write the report in Failed() and Passed()

https://github.com/RamenDR/ramenctl/compare/c2cf7be894bcaa5870def678466535025713378d..b7f254c8a8310df45e119b76ac83e72e7a90a55e

Setup and Cleanup are are important concepts for a test command. Make them a method on the command type. This cleans up the code, and will make it easier to update the command report in the next changes. Signed-off-by: Nir Soffer <[email protected]>

Signed-off-by: Nir Soffer <[email protected]>

This is useful to report multiple errors, required when running multiple tests, or when we need to report non-fatal errors that do not fail the entire run. Signed-off-by: Nir Soffer <[email protected]>

The report package provides the common part of a report, used by all reports. Specific reports (e.g. test report) will add more info to the report. The common report includes information about the host and about ramenctl. This should aid in debugging issues related to the host or to specific ramenctl build. Example report yaml: host: arch: arm64 cpus: 12 os: darwin name: test-run ramenctl: commit: 78fc3c5420fb1c78f006a0dadd0c82bc9cf14883 version: v0.1.0-3-g78fc3c5 Signed-off-by: Nir Soffer <[email protected]>

Add Config and Status fields to Test. When a test fails, we mark the test as failed. The Config and Status will be used to report the test result. Signed-off-by: Nir Soffer <[email protected]>

The test.Report adds tests specific info, describing a test run or clean. The report adds: - Name: "test-run" or "test-clean" - Status: "passed" if all tests passed, "failed" if setup failed, or some tests failed. - Summary: Count of passed, failed, and skipped tests - Steps: list of steps ran. We have 3 steps: - Setup: run once before the tests during a test run command - Tests: run once after setup during a test run command - Cleanup: run once after the tests during a test clean command. Example report for test run: host: arch: arm64 cpus: 12 os: darwin name: test-run ramenctl: commit: 78fc3c5420fb1c78f006a0dadd0c82bc9cf14883 version: v0.1.0-3-g78fc3c5 status: passed steps: - name: setup status: passed - name: tests status: passed tests: - deployer: appset pvcSpec: rbd status: passed workload: deploy summary: failed: 0 passed: 1 skipped: 0 Signed-off-by: Nir Soffer <[email protected]>

Reports are stored in the command output directory using "command-name.yaml". This makes it easy to create reports from any command. I'm thinking about keeping all reports in single file as a stream of yaml documents, but let's start with something simple. Signed-off-by: Nir Soffer <[email protected]>

This matches better the command behavior. For test run: 1. Run setup 2. Run tests For test clean: 1. Clean tests 2. Run cleanup The natural place for running or cleaning a test seems to be the test command. already used to run the setup and cleanup steps. This will also make it easier to integrate the test report in the next step. This makes is easier to handle test failure and support running multiple tests in parallel. Signed-off-by: Nir Soffer <[email protected]>

Previously if a test step (setup, cleanup, test) failed, we returned an error to signal the failure. This aborts the entire test and the top level command code was logging the error to the console and exit. This scheme does not work for complicated flow such as running multiple test in parallel, of for handling multiple errors (e.g. failure to write a test report after test step failed). Change the interface so failed step log the error to the console and logfile and return false, signaling that the operation failed and we need to abort the test. Previously we log the test success message in the top level command, but this does not allow logging useful info when tests are finished, such as number of passed and failed tests. Move the log to the test package. When a test complete, it returns either a Command.Failed() error, or call Command.Passed(). These methods will be used to write the test report and report more info about the test. Signed-off-by: Nir Soffer <[email protected]>

Since both run and clean commands need to store a report create a report when creating a command. Modify Setup(), Cleanup() and RunTest() to update the report when the operation completes. Modify Failed() and Passed() to write the report to the output directory, and report test summary in the failure or success message. Signed-off-by: Nir Soffer <[email protected]>

nirs · 2025-03-21T18:03:32Z

Changes in last version:

Unify Command name in methods (c *Command)
Add missing console.Error() in Setup() and Cleanup() failure (bad rebase)

nirs force-pushed the report branch 4 times, most recently from 59aef42 to 743d16a Compare March 20, 2025 22:23

nirs marked this pull request as ready for review March 20, 2025 22:24

nirs force-pushed the report branch from 743d16a to 697a53f Compare March 20, 2025 22:27

nirs requested review from raghavendra-talur, rakeshgm and parikshithb March 20, 2025 22:27

nirs force-pushed the report branch 2 times, most recently from 46406bc to c2cf7be Compare March 20, 2025 23:32

parikshithb approved these changes Mar 21, 2025

View reviewed changes

nirs force-pushed the report branch from c2cf7be to b7f254c Compare March 21, 2025 16:31

rakeshgm approved these changes Mar 21, 2025

View reviewed changes

nirs added 10 commits March 21, 2025 19:51

Move Setup and Cleanup to Command

069ea73

Setup and Cleanup are are important concepts for a test command. Make them a method on the command type. This cleans up the code, and will make it easier to update the command report in the next changes. Signed-off-by: Nir Soffer <[email protected]>

Fix missing argument in error

c265e6f

Signed-off-by: Nir Soffer <[email protected]>

Add console.Error()

2ee3597

This is useful to report multiple errors, required when running multiple tests, or when we need to report non-fatal errors that do not fail the entire run. Signed-off-by: Nir Soffer <[email protected]>

Keep test status and configuration

743886f

Add Config and Status fields to Test. When a test fails, we mark the test as failed. The Config and Status will be used to report the test result. Signed-off-by: Nir Soffer <[email protected]>

nirs force-pushed the report branch from b7f254c to c0fc7a2 Compare March 21, 2025 18:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add test report #48

Add test report #48

nirs commented Mar 19, 2025 •

edited

Loading

parikshithb left a comment

nirs commented Mar 21, 2025 •

edited

Loading

nirs commented Mar 21, 2025

Add test report #48

Are you sure you want to change the base?

Add test report #48

Conversation

nirs commented Mar 19, 2025 • edited Loading

Example test run command

Example test clean command

Example report

Report contents

test-run.yaml

test-clean.yaml

Report tarball

parikshithb left a comment

Choose a reason for hiding this comment

nirs commented Mar 21, 2025 • edited Loading

nirs commented Mar 21, 2025

nirs commented Mar 19, 2025 •

edited

Loading

nirs commented Mar 21, 2025 •

edited

Loading