Skip to content

Add ONT BAM and bedMethyl test data: HG002 GIAB 10-read subset (PAW70337) - #1969

Open
sahuno wants to merge 1 commit into
nf-core:modulesfrom
sahuno:add-giab-ont-bam-testdata
Open

Add ONT BAM and bedMethyl test data: HG002 GIAB 10-read subset (PAW70337)#1969
sahuno wants to merge 1 commit into
nf-core:modulesfrom
sahuno:add-giab-ont-bam-testdata

Conversation

@sahuno

@sahuno sahuno commented Apr 6, 2026

Copy link
Copy Markdown

Summary

Adds 5 test data files derived from the GIAB HG002 ONT run PAW70337 (5kHz, R10.4.1), companion to the pod5 file merged in #1968.

New files

Unaligned BAM — raw dorado basecaller output, no reference:

  • data/genomics/homo_sapiens/nanopore/bam/HG002_PAW70337_giab_10reads.unaligned.bam (178 KB)

Aligned sorted BAM + index — coordinate-sorted alignment to hg38 via minimap2:

  • data/genomics/homo_sapiens/nanopore/bam/HG002_PAW70337_giab_10reads.aligned.sorted.bam (314 KB)
  • data/genomics/homo_sapiens/nanopore/bam/HG002_PAW70337_giab_10reads.aligned.sorted.bam.bai (566 KB)

bedMethyl + tabix index — modkit pileup output (5mCG+5hmCG), bgzipped:

  • data/genomics/homo_sapiens/nanopore/methylation/HG002_PAW70337_giab_10reads.aligned.sorted.bedmethyl.gz (69 KB)
  • data/genomics/homo_sapiens/nanopore/methylation/HG002_PAW70337_giab_10reads.aligned.sorted.bedmethyl.gz.tbi (1.7 KB)

Modules that will use these files

File Module(s)
unaligned.bam dorado/summary, dorado/trim, dorado/correct
aligned.sorted.bam + .bai modkit/pileup, dorado/basecaller functional test
bedmethyl.gz + .tbi modkit/localize, modkit/localize/plot, modkit/pileup/plot

Source

s3://ont-open-data/giab_2025.01/flowcells/HG002/PAW70337/pod5/PAW70337_66b2eea5_de8117b1_33.pod5
  • Sample: HG002 (NIST/Genome in a Bottle reference sample, public domain)
  • Chemistry: 5kHz, R10.4.1 (PAW70337, released 2025-01-14)
  • Subset: 10 reads — basecalled with dorado 1.4.0, aligned with minimap2, methylation called with modkit 0.6.1

Test plan

  • BAM opens with samtools view
  • BAI index validates with samtools idxstats
  • bedMethyl decompresses and tabix queries correctly
  • All files used in passing nf-test stub tests for the modules above

🤖 Generated with Claude Code

@sahuno @claude
Add ONT BAM and bedMethyl test data: HG002 GIAB 10-read subset (PAW70…
6a35295 
…337)

Adds 5 files derived from the GIAB HG002 ONT run PAW70337 (5kHz, R10.4.1),
companion to the pod5 file added in nf-core#1968.

New files:
- nanopore/bam/HG002_PAW70337_giab_10reads.unaligned.bam (178 KB)
  Raw dorado basecaller output, no reference alignment.
  Used by: dorado/summary, dorado/trim, dorado/correct

- nanopore/bam/HG002_PAW70337_giab_10reads.aligned.sorted.bam (314 KB)
- nanopore/bam/HG002_PAW70337_giab_10reads.aligned.sorted.bam.bai (566 KB)
  Coordinate-sorted alignment to hg38 via minimap2.
  Used by: modkit/pileup, modkit/dmr, dorado/basecaller functional test

- nanopore/methylation/HG002_PAW70337_giab_10reads.aligned.sorted.bedmethyl.gz (69 KB)
- nanopore/methylation/HG002_PAW70337_giab_10reads.aligned.sorted.bedmethyl.gz.tbi (1.7 KB)
  modkit pileup output (5mCG+5hmCG), bgzipped + tabix indexed.
  Used by: modkit/localize, modkit/localize/plot, modkit/pileup/plot

Source: s3://ont-open-data/giab_2025.01/flowcells/HG002/PAW70337/pod5/
Sample: HG002 (NIST/Genome in a Bottle), public domain data.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@sahuno sahuno mentioned this pull request Apr 8, 2026
Open
11 tasks
@sahuno sahuno mentioned this pull request Apr 18, 2026
Open
11 tasks
@sahuno

sahuno commented Apr 18, 2026

Copy link
Copy Markdown
Author

Hey @dialvarezs 👋 , when you have a moment, could you pls review data inclusion pr?

This PR adds the unaligned/aligned HG002 BAM + bedMethyl test data (same GIAB 10-read subset as #1968, which you approved). It's the test-data dependency for two downstream module PRs:

Thanks!

dialvarezs
dialvarezs reviewed Apr 28, 2026
View reviewed changes

@dialvarezs dialvarezs left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @sahuno, sorry for the delay on this review.

Looking at the individual files, I think the unaligned BAM and aligned BAM + BAI make sense as reusable fixtures, especially if we want to avoid chaining through GPU-heavy dorado/basecaller steps.

The one I’m less sure about is the pre-generated bedmethyl.gz + .tbi, since that is already a downstream modkit pileup output. Would it be feasible to generate this via setup() from the aligned BAM for the other modkit tests instead? That way, we could preserve some chaining between dependent modules and help catch breaking changes more effectively over time.

@dialvarezs

dialvarezs commented Apr 28, 2026

Copy link
Copy Markdown
Member

Also, what do you mean by "dorado/basecaller functional test`?

sahuno added a commit to sahuno/modules that referenced this pull request Sep 6, 2026
@sahuno @claude
fix(dorado/aligner): use existing test data, correct summary output, …
b703eef 
…skip conda

All six nf-tests were failing because the fixture they referenced,
`HG002_PAW70337_giab_10reads.unaligned.bam`, was never merged into
nf-core/test-datasets (it is still pending in nf-core/test-datasets#1969), so
`checkIfExists: true` threw before the process ever ran.

Tests now use `test2.sorted.bam`, which is already merged and maps to the same
chr22 subset reference. `test.sorted.phased.bam` cannot be used: each of its
records carries two RG tags (one from minimap2 `-R`, one from the original
dorado basecall) while its header declares only one. dorado preserves both, so
the output BAM trips htsjdk's READ_GROUP_NOT_FOUND check inside `getReadsMD5()`.

Other fixes found while verifying against dorado 1.4.0:

- The `summary` output could never be produced. `--emit-summary` writes a
  fixed-name `sequencing_summary.txt`, not `*_summary.tsv`. The glob is
  corrected and the script renames the file to `<prefix>.sequencing_summary.txt`
  so outputs stay per-sample. The test now sets `--emit-summary` and asserts the
  channel is populated.
- Removed the dead `versions.yml` heredoc from the stub: this module reports
  versions through the `versions` topic, so nothing consumed that file.
- Removed a no-op `sed 's/^//'` from the version eval.
- The real test now actually computes the reads-only MD5 its comment promised,
  via nft-bam's `getReadsMD5()`, and snapshots it under a `versions` key so the
  `test_snap_versions` lint check passes.
- Corrected the `.fai` input ontology, which pointed at the EDAM term for BED.

Added `modules/nf-core/dorado` to the conda skip list, matching how parabricks
is handled: dorado is not on bioconda (ONTPL licence) so the conda profile can
never resolve it. docker and singularity are unaffected.

Remaining known failure is `container_links`: nf-core lint only accepts
`quay.io` or `community.wave.seqera.io/library` prefixes, and dorado is
published only on Docker Hub. Flagged for maintainer input.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E7RSvpouxbMBdimYSwJ5ZP
@sahuno

sahuno commented Sep 6, 2026

Copy link
Copy Markdown
Author

@dialvarezs thanks for the review, and apologies for the long silence.

On the bedmethyl files — I agree, drop them. You are right that a
pre-generated modkit pileup output is a downstream artefact rather than a
fixture, and generating it via setup() keeps the chaining between dependent
modules honest. I have since done exactly that in two module PRs:

So this PR can be reduced to just the unaligned BAM + aligned BAM + BAI, which
is the part you said made sense. Happy to force-push that reduction — just
confirm and I will drop the two bedmethyl.gz/.tbi files.

On "dorado/basecaller functional test" — sorry, that was unclear wording on
my part. I meant the non-stub, GPU-tagged tests in nf-core/modules#11122, which
need a real pod5 rather than a stub, hence the unaligned BAM here for the
downstream alignment tests.

One update worth flagging: nf-core/modules#11222 (dorado/aligner) no longer
depends on this PR at all. Its tests were failing because they referenced
HG002_PAW70337_giab_10reads.unaligned.bam from here before it was merged, so I
switched them to the already-merged test2.sorted.bam. That PR is now green
independently, so this one is no longer blocking it.

@dialvarezs

dialvarezs commented Sep 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

just confirm and I will drop the two bedmethyl.gz/.tbi files.

Confirmed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dialvarezs dialvarezs dialvarezs left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@sahuno @dialvarezs