Use ctest, parallelize tests, report coverage #538
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sbozzolo
added
test
Sbozzolo
force-pushed
the
gbozzola/test_coverage
branch
4 times, most recently
from
a122959 to
aea4162
Compare
Sbozzolo
commented
Oct 23, 2025
| } // namespace | ||
|
|
||
| TEST_CASE("2D libCEED Operators", "[libCEED][Serial][Parallel][GPU]") | ||
| TEST_CASE("2D libCEED Operators", "[libCEED][Serial][Parallel]") |
Member
Author
There was a problem hiding this comment.
Removing the GPU tag because of #516 (ultimately, it is likely that these tests were never meant to be run on GPU as-is)
Sbozzolo
force-pushed
the
gbozzola/test_coverage
branch
7 times, most recently
from
a631499 to
faa368e
Compare
This commit introduces a few improvements to the test framework: * Support for `ctest` Tests can now be run with: ``` ctest -j $(nproc) ``` This will automatically take care of running all the tests, including the MPI tests (with two process) and the GPU tests (if applicable). Tests run in this way are not run in parallel. Note, OpenMP threads are not accounted here, so the total number of cores used is $(nproc) * NUM_OMP_THREADS * New test tag `NoConcurrent` and new fixture `TempDirFixture` Since tests can now be run in parallel, we need to tag tests that cannot be run concurrently (e.g., because they used shared resources). A common use case of `NoConcurrent` is for file-system operations. However, this is a better way to deal with this problem: ensuring isolation and clean up of tests. To help with this, I added a new fixture, `TempDirFixture`, that automatically creates and destroy testing directories. * script to compute coverage `measure-test-coverage` is a new script that uses the ctest capabilities and lcov to distill coverage information and produce html reports. `measure-test-coverage` can be used to: 1. produce the coverage.info objects by running the tests 2. merge different coverage.info objects 3. produce the html report from the coverage.info files When run locally, `measure-test-coverage report` will probably cover most of the use cases. The need to handle different coverage.info objects might become relevant in the future for CI (if test coverage information is split across different workers) * Coverage is now computed as part of CI: With all the previous advancements, we can now compute coverage as part of CI and produce a report. The report can be downloaded from the job page on GitHub actions.
Sbozzolo
force-pushed
the
gbozzola/test_coverage
branch
from
faa368e to
7675acc
Compare
Sbozzolo
commented
Oct 23, 2025
|
|
||
| permissions: | ||
| contents: read | ||
| contents: write |
Member
Author
There was a problem hiding this comment.
Now we push to gh-pages
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit introduces a few improvements to the test framework:
Support for
ctestTests can now be run with:
This will automatically take care of running all the tests, including the MPI tests (with two process) and the GPU tests (if applicable).
Tests run in this way are not run in parallel.
Note, OpenMP threads are not accounted here, so the total number of cores used is $(nproc) * NUM_OMP_THREADS
New test tag
NoConcurrentand new fixtureTempDirFixtureSince tests can now be run in parallel, we need to tag tests that cannot be run concurrently (e.g., because they used shared resources).
A common use case of
NoConcurrentis for file-system operations. However, this is a better way to deal with this problem: ensuring isolation and clean up of tests. To help with this, I added a new fixture,TempDirFixture, that automatically creates and destroy testing directories.script to compute coverage
measure-test-coverageis a new script that uses the ctest capabilities and lcov to distill coverage information and produce html reports.measure-test-coveragecan be used to:When run locally,
measure-test-coverage reportwill probably cover most of the use cases. The need to handle different coverage.info objects might become relevant in the future for CI (if test coverage information is split across different workers)Coverage is now computed as part of CI:
With all the previous advancements, we can now compute coverage as part of CI and produce a report. The report can be downloaded from the job page on GitHub actions and is uploaded to
gh-pages:https://awslabs.github.io/palace/previews/PR538/coverage/
I also added a clickable link to the summary of the GitHub job:
