Re-test and label/exclude modules with failing tests #4016
Description
Problem
When a widely-used module (e.g., untar, samtools/index) is modified, it triggers tests for all dependent modules and subworkflows. This frequently surfaces pre-existing test failures in modules that haven't been tested in a while, blocking PRs that are unrelated to those failures.
This creates a frustrating situation where contributors end up debugging and fixing other people's broken tests just to get their own changes merged, or requiring maintainers to override CI failures.
Recent example: nf-core/modules#9796 - a topic channel migration for 5 modules triggered failures in galah, kraken subworkflows, and others that were already broken.
Proposed Solution
As agreed in the nf-core maintainers meeting, implement a periodic CI job (weekly/monthly?) that:
- Runs all module/subworkflow tests regardless of changes
- Labels modules with failing tests (e.g.,
tests-failing,needs-fix) - Optionally excludes known-failing modules from PR CI runs until fixed
This would:
- Surface broken tests proactively rather than when someone touches a dependency
- Make it clear which modules need attention
- Prevent unrelated failures from blocking PRs
- Provide a "health dashboard" for the modules repository
Additional Considerations
- Labels could trigger notifications to module maintainers
- A scheduled issue could be auto-created listing all failing modules
- Consider a "quarantine" mechanism where known-failing tests are skipped in PR CI but tracked separately