-
Notifications
You must be signed in to change notification settings - Fork 286
Add git-bisect script/workflow and generic single-target build/test script #5728
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
816cd69 to
1335ccd
Compare
8fe9ce0 to
044bcf6
Compare
910ad6d to
1c1eb63
Compare
|
Most recent run of the bisect workflow: https://github.com/NVIDIA/cccl/actions/runs/17365589071/attempts/1#summary-49291821571 |
1c1eb63 to
42074d2
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR adds comprehensive git bisection tooling and targeted build/test utilities to help developers identify regression commits and efficiently test specific targets. It provides both local and CI-based workflow solutions for automated regression testing.
- Unified build/test script for targeted execution of CMake, ninja, ctest, and lit targets
- Git bisection wrapper that automates regression testing across git history
- GitHub Actions workflow for remote bisection with browser-based job initiation
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
docs/cccl/development/index.rst |
Adds reference to new build and bisect tools documentation |
docs/cccl/development/build_and_bisect_tools.rst |
Comprehensive documentation for the new build/test and git bisect utilities |
ci/util/git_bisect.sh |
Main git bisection script that wraps git bisect around the build/test helper |
ci/util/build_and_test_targets.sh |
Generic script for configuring, building, and testing specific CMake/lit targets |
.github/workflows/git-bisect.yml |
GitHub Actions workflow for running bisection jobs remotely |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
42074d2 to
72fb391
Compare
🟨 CI finished in 1h 15m: Pass: 97%/210 | Total: 1d 17h | Avg: 11m 44s | Max: 38m 02s | Hits: 88%/312237
|
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| CCCL Packaging | |
| libcu++ | |
| CUB | |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| python | |
| CCCL C Parallel Library | |
| Catch2Helper | |
| NVBench Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| +/- | CCCL Packaging |
| +/- | libcu++ |
| +/- | CUB |
| +/- | Thrust |
| +/- | CUDA Experimental |
| +/- | stdpar |
| +/- | python |
| +/- | CCCL C Parallel Library |
| +/- | Catch2Helper |
| +/- | NVBench Helper |
🏃 Runner counts (total jobs: 210)
| # | Runner |
|---|---|
| 128 | linux-amd64-cpu16 |
| 23 | windows-amd64-cpu16 |
| 17 | linux-amd64-gpu-l4-latest-1 |
| 12 | linux-arm64-cpu16 |
| 11 | linux-amd64-gpu-rtx2080-latest-1 |
| 10 | linux-amd64-gpu-h100-latest-1 |
| 6 | linux-amd64-gpu-rtxa6000-latest-1 |
| 3 | linux-amd64-gpu-rtx4090-latest-1 |
- build_and_test_targets.sh: reproducible single-target configure/build/test (ninja, ctest, lit) - git_bisect.sh: wrapper to automate regression testing with minimal targets - Workflow/Bisect: GitHub Actions job to run bisect remotely and generate Markdown summaries
72fb391 to
b696ce9
Compare
🟨 CI finished in 1h 10m: Pass: 97%/210 | Total: 1d 13h | Avg: 10m 47s | Max: 40m 55s | Hits: 95%/312237
|
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| CCCL Packaging | |
| libcu++ | |
| CUB | |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| python | |
| CCCL C Parallel Library | |
| Catch2Helper | |
| NVBench Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| +/- | CCCL Packaging |
| +/- | libcu++ |
| +/- | CUB |
| +/- | Thrust |
| +/- | CUDA Experimental |
| +/- | stdpar |
| +/- | python |
| +/- | CCCL C Parallel Library |
| +/- | Catch2Helper |
| +/- | NVBench Helper |
🏃 Runner counts (total jobs: 210)
| # | Runner |
|---|---|
| 128 | linux-amd64-cpu16 |
| 23 | windows-amd64-cpu16 |
| 17 | linux-amd64-gpu-l4-latest-1 |
| 12 | linux-arm64-cpu16 |
| 11 | linux-amd64-gpu-rtx2080-latest-1 |
| 10 | linux-amd64-gpu-h100-latest-1 |
| 6 | linux-amd64-gpu-rtxa6000-latest-1 |
| 3 | linux-amd64-gpu-rtx4090-latest-1 |
🟨 CI finished in 1h 12m: Pass: 99%/210 | Total: 1d 17h | Avg: 11m 43s | Max: 52m 48s | Hits: 94%/323908
|
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| CCCL Packaging | |
| libcu++ | |
| CUB | |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| python | |
| CCCL C Parallel Library | |
| Catch2Helper | |
| NVBench Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| +/- | CCCL Packaging |
| +/- | libcu++ |
| +/- | CUB |
| +/- | Thrust |
| +/- | CUDA Experimental |
| +/- | stdpar |
| +/- | python |
| +/- | CCCL C Parallel Library |
| +/- | Catch2Helper |
| +/- | NVBench Helper |
🏃 Runner counts (total jobs: 210)
| # | Runner |
|---|---|
| 128 | linux-amd64-cpu16 |
| 23 | windows-amd64-cpu16 |
| 17 | linux-amd64-gpu-l4-latest-1 |
| 12 | linux-arm64-cpu16 |
| 11 | linux-amd64-gpu-rtx2080-latest-1 |
| 10 | linux-amd64-gpu-h100-latest-1 |
| 6 | linux-amd64-gpu-rtxa6000-latest-1 |
| 3 | linux-amd64-gpu-rtx4090-latest-1 |
🟩 CI finished in 4h 14m: Pass: 100%/210 | Total: 1d 16h | Avg: 11m 42s | Max: 52m 48s | Hits: 94%/325821
|
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| CCCL Packaging | |
| libcu++ | |
| CUB | |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| python | |
| CCCL C Parallel Library | |
| Catch2Helper | |
| NVBench Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| +/- | CCCL Packaging |
| +/- | libcu++ |
| +/- | CUB |
| +/- | Thrust |
| +/- | CUDA Experimental |
| +/- | stdpar |
| +/- | python |
| +/- | CCCL C Parallel Library |
| +/- | Catch2Helper |
| +/- | NVBench Helper |
🏃 Runner counts (total jobs: 210)
| # | Runner |
|---|---|
| 128 | linux-amd64-cpu16 |
| 23 | windows-amd64-cpu16 |
| 17 | linux-amd64-gpu-l4-latest-1 |
| 12 | linux-arm64-cpu16 |
| 11 | linux-amd64-gpu-rtx2080-latest-1 |
| 10 | linux-amd64-gpu-h100-latest-1 |
| 6 | linux-amd64-gpu-rtxa6000-latest-1 |
| 3 | linux-amd64-gpu-rtx4090-latest-1 |
…cript (NVIDIA#5728) * Add targeted build/test + bisect scripts and workflow - build_and_test_targets.sh: reproducible single-target configure/build/test (ninja, ctest, lit) - git_bisect.sh: wrapper to automate regression testing with minimal targets - Workflow/Bisect: GitHub Actions job to run bisect remotely and generate Markdown summaries * Pass `--rm` to docker run when using launch.sh * Helpful defaults for browser UI, allow workflow to run in forks.
Summary
Provides a single script for targeted builds/tests, giving developers and agentic AIs a unified, efficient interface for testing changes / reproducing issues. Supports ninja, ctest, and lit targets. A single command can now configure, build, and test specific targets in a specific devcontainer in the CCCL repo. Enables developers to quickly share reproducible, single-target build/test instructions.
The new targeted build/test script is used to implement a CCCL-specific
git bisectwrapper script that automates regression testing. A single command will launch a devcontainer and startgit bisect run, automatically testing a minimal set of build/test targets across the git history.A new Github Actions workflow has been added that wraps the new
git bisectscript. This allows bisection jobs to be started from the Github Actions UI in the browser, producing a markdown summary detailing the findings.Details and examples are found below.
build_and_test_targets.shci/util/build_and_test_targets.shconfigures, builds, and tests any set of CMake / lit targets.Options
--preset <name>: use a CMake preset.--cmake-options <str>: extra flags for the preset. Ignored if--configure-overrideis given.--configure-override <cmd>: run a custom configure command instead of a preset.--build-targets <targets>: space‑separated Ninja targets. Nothing builds unless specified.--ctest-targets <regex>: space‑separated CTest-Rpatterns. Nothing runs unless specified.--lit-precompile-tests <paths>: space‑separated libcudacxx lit test paths to precompile (no run). Paths are relative tolibcudacxx/test/libcudacxx/.--lit-tests <paths>: space‑separated libcudacxx lit test paths to execute. Paths are relative tolibcudacxx/test/libcudacxx/.--custom-test-cmd <cmd>: arbitrary command executed after build/tests.--configure-overridesupersedes--presetand--cmake-options.Pair with
.devcontainer/launch.sh -dto choose a CUDA toolkit and host compiler:.devcontainer/launch.sh -d --cuda <ver> --host <compiler> -- <script>Examples
Build a single CUB test locally:
ci/util/build_and_test_targets.sh \ --preset cub-cpp20 \ --build-targets "cub.cpp20.test.iterator"Build the same test for SM90 using a CMake option:
Build the test for SM90 with a configure override:
Build and run a single CUB test locally:
Build and run a single CUB test in a devcontainer with specific CTK and host compiler:
.devcontainer/launch.sh -d --cuda 12.3 --host gcc12 --gpus all -- \ ci/util/build_and_test_targets.sh \ --preset cub-cpp20 \ --build-targets "cub.cpp20.test.iterator" \ --ctest-targets "cub.cpp20.test.iterator"Precompile the libcudacxx lit suite:
Precompile a single libcudacxx lit test (no execution):
ci/util/build_and_test_targets.sh \ --preset libcudacxx-cpp20 \ --lit-precompile-tests \ "std/algorithms/alg.nonmodifying/alg.any_of/any_of.pass.cpp"Execute one or more libcudacxx lit tests:
ci/util/build_and_test_targets.sh \ --preset libcudacxx-cpp20 \ --lit-tests \ "std/algorithms/alg.nonmodifying/alg.any_of/any_of.pass.cpp"git_bisect.shci/util/git_bisect.shwrapsgit bisectaround the build/test helper.Options
All options from
build_and_test_targets.sh, plus:--good-ref <rev>: known good commit, tag, or branch.-Ndfor "N days ago".--bad-ref <rev>: known bad commitorigin/main.-Ndfor "N days ago".--summary-file <path>: write Markdown results.Examples
Local CUB bisection from last release to origin/main:
Devcontainer CUB bisection over last week:
.devcontainer/launch.sh -d --cuda 12.3 --host gcc12 --gpus all -- \ ci/util/git_bisect.sh \ --preset cub-cpp20 \ --build-targets "cub.cpp20.test.iterator" \ --ctest-targets "cub.cpp20.test.iterator" \ --good-ref -7dCompute-sanitizer example, knowing regression was 3-4 weeks ago:
.devcontainer/launch.sh -d --cuda 12.9 --host gcc13 --gpus all \ --env CCCL_TEST_MODE=compute-sanitizer-initcheck \ --env C2H_SEED_COUNT_OVERRIDE=1 \ -- ci/util/git_bisect.sh \ --preset "cub-cpp20" \ --build-targets "cub.cpp20.test.iterator" \ --ctest-targets "cub.cpp20.test.iterator" \ --good-ref -28d \ --bad-ref -21dWorkflow/Bisect
GitHub Actions offers a "Workflow/Bisect" job that runs
git_bisect.shremotely.From the repository page:
Example Output
🔎 Bisect Result
thrust::copyinthrust::uninitialized_copy[_n]in CUDA system when possible (Usethrust::copyinthrust::uninitialized_copy[_n]in CUDA system when possible #5181)thrust::copyinthrust::uninitialized_copy[_n]in CUDA system when possible #5181♻️ Reproduction Steps
.devcontainer/launch.sh \ --cuda 12.9 --host gcc13 --env CCCL_TEST_MODE=compute-sanitizer-initcheck --env C2H_SEED_COUNT_OVERRIDE=1 --gpus all \ -- \ ./ci/util/build_and_test_targets.sh \ --preset 'cub-cpp20' \ --cmake-options '-DCMAKE_CUDA_FLAGS=-lineinfo -DCMAKE_CUDA_ARCHITECTURES=86' \ --configure-override '' \ --build-targets 'cub.cpp20.test.iterator' \ --ctest-targets 'cub.cpp20.test.iterator' \ --lit-precompile-tests '' \ --lit-tests '' \ --custom-test-cmd ''ℹ️ Commit Details