rnaSPADES error singularity image

[shellywanamaker]

Description of the bug

I have not been able to get rnaSPADES to complete successfully and have encountered the following errors in order of most recent:

https://gannet.fish.washington.edu/metacarcinus/USDA_MetaOmics/Cgigas_denovotranscript/20241104/old_reports/attempt3/nf_report.html

https://gannet.fish.washington.edu/metacarcinus/USDA_MetaOmics/Cgigas_denovotranscript/20241104/old_reports/attempt2/nf_report.html

is this a memory issue?

Command used and terminal output

nextflow run nf-core/denovotranscript -c /gscratch/srlab/strigg/bin/uw_hyak_srlab.config --input /gscratch/scrubbed/strigg/analyses/20240925/samplesheet/samplesheet.csv --outdir /gscratch/scrubbed/strigg/analyses/20241104_denovo '--extra_fastp_args=--trim_front1 10 --trim_front2 10' --remove_ribo_rna --assemblers rnaspades -resume -with-report nf_report.html -with-trace -with-timeline nf_timeline.html

Workflow execution completed unsuccessfully!
The exit status of the task that caused the workflow execution to fail was: 250.

The full error message was:

Error executing process > 'NFCORE_DENOVOTRANSCRIPT:DENOVOTRANSCRIPT:SPADES (pooled_reads)'

Caused by:
Process NFCORE_DENOVOTRANSCRIPT:DENOVOTRANSCRIPT:SPADES (pooled_reads) terminated with an error exit status (250)

Command executed:

spades.py
--rna
--threads 12
--memory 72

-1 pooled_reads_1.merged.fastq.gz -2 pooled_reads_2.merged.fastq.gz

-o ./
mv spades.log pooled_reads.spades.log

if [ -f scaffolds.fasta ]; then
mv scaffolds.fasta pooled_reads.scaffolds.fa
gzip -n pooled_reads.scaffolds.fa
fi
if [ -f contigs.fasta ]; then
mv contigs.fasta pooled_reads.contigs.fa
gzip -n pooled_reads.contigs.fa
fi
if [ -f transcripts.fasta ]; then
mv transcripts.fasta pooled_reads.transcripts.fa
gzip -n pooled_reads.transcripts.fa
fi
if [ -f soft_filtered_transcripts.fasta ]; then
mv soft_filtered_transcripts.fasta pooled_reads.soft_filtered_transcripts.fa
gzip -n pooled_reads.soft_filtered_transcripts.fa
fi
if [ -f hard_filtered_transcripts.fasta ]; then
mv hard_filtered_transcripts.fasta pooled_reads.hard_filtered_transcripts.fa
gzip -n pooled_reads.hard_filtered_transcripts.fa
fi
if [ -f assembly_graph_with_scaffolds.gfa ]; then
mv assembly_graph_with_scaffolds.gfa pooled_reads.assembly.gfa
gzip -n pooled_reads.assembly.gfa
fi

if [ -f gene_clusters.fasta ]; then
mv gene_clusters.fasta pooled_reads.gene_clusters.fa
gzip -n pooled_reads.gene_clusters.fa
fi

cat <<-END_VERSIONS > versions.yml
"NFCORE_DENOVOTRANSCRIPT:DENOVOTRANSCRIPT:SPADES":
spades: $(spades.py --version 2>&1 | sed -n 's/^.*SPAdes genome assembler v//p')
END_VERSIONS

Command exit status:
250

Command output:
2:58:16.891 49G / 57G INFO Simplification (parallel_processing.hpp : 171) Relative coverage component remover triggered 0 times
2:58:16.954 49G / 57G INFO Simplification (parallel_processing.hpp : 168) Running Complex bulge remover
2:58:29.318 49G / 57G INFO Simplification (parallel_processing.hpp : 171) Complex bulge remover triggered 0 times
2:58:29.382 49G / 57G INFO Simplification (parallel_processing.hpp : 168) Running Tip clipper
2:58:29.443 49G / 57G INFO Simplification (parallel_processing.hpp : 171) Tip clipper triggered 0 times
2:58:29.507 49G / 57G INFO Simplification (parallel_processing.hpp : 168) Running Final tip clipper
2:58:29.568 49G / 57G INFO Simplification (parallel_processing.hpp : 171) Final tip clipper triggered 0 times
2:58:29.631 49G / 57G INFO Simplification (parallel_processing.hpp : 168) Running Bulge remover
2:58:29.690 49G / 57G INFO Simplification (parallel_processing.hpp : 171) Bulge remover triggered 0 times
2:58:29.750 49G / 57G INFO Simplification (parallel_processing.hpp : 168) Running Final bulge remover
2:58:29.809 49G / 57G INFO Simplification (parallel_processing.hpp : 171) Final bulge remover triggered 0 times
2:58:29.869 49G / 57G INFO Simplification (parallel_processing.hpp : 168) Running AT Tips
2:58:29.928 49G / 57G INFO Simplification (parallel_processing.hpp : 171) AT Tips triggered 0 times
2:58:29.987 49G / 57G INFO General (simplification.cpp : 348) Disrupting self-conjugate edges
2:58:32.102 49G / 57G INFO Simplification (parallel_processing.hpp : 168) Running Removing isolated edges
2:58:35.568 49G / 57G INFO Simplification (parallel_processing.hpp : 171) Removing isolated edges triggered 2140990 times
2:58:36.049 49G / 57G INFO General (simplification.cpp : 508) After simplification:
2:58:36.110 49G / 57G INFO General (simplification.cpp : 509) Average coverage = 194.651
2:58:36.669 49G / 57G INFO General (simplification.cpp : 510) Total length = 202779300
2:58:37.195 49G / 57G INFO General (simplification.cpp : 511) Median edge length: 356
2:58:37.261 49G / 57G INFO General (simplification.cpp : 512) Edges: 2070079
2:58:37.321 49G / 57G INFO General (simplification.cpp : 513) Vertices: 2424436
2:58:37.382 49G / 57G INFO StageManager (stage.cpp : 189) STAGE == Mismatch Correction (id: mismatch_correction)
2:58:37.442 49G / 57G INFO General (graph_pack_helpers.cpp : 44) Index refill
2:58:37.502 49G / 57G INFO General (edge_index.hpp : 175) Using small index (max_id = 225479915)
2:58:39.137 49G / 57G INFO K-mer Index Building (kmer_index_builder.hpp : 410) Building perfect hash indices
2:58:56.297 49G / 57G INFO K-mer Index Building (kmer_index_builder.hpp : 446) Index built. Total 202779278 kmers, 146680906 bytes occupied (5.78682 bits per kmer).
2:58:56.916 50G / 57G INFO General (edge_index_builders.hpp : 253) Collecting edge information from graph, this takes a while.
2:59:04.393 50G / 57G INFO General (graph_pack_helpers.cpp : 54) Normalizing k-mer map. Total 662334904 kmers to process
mimalloc: error: unable to allocate OS memory (5305794560 bytes, error code: 12 [Cannot allocate memory], address: (nil), large only: 0, allow large: 1)
mimalloc: process info:
elapsed: 10841.233 s
process: user: 94959.883 s, system: 1410.633 s, faults: 351352
rss: current: 126547222528, peak: 70665609216
commit: current: 126547222528, peak: 126547222528

== Error == system call for: "['/usr/local/bin/spades-core', '/gscratch/scrubbed/srlab/nxf.si5WitkIa9/K43/configs/config.info', '/gscratch/scrubbed/srlab/nxf.si5WitkIa9/K43/configs/mda_mode.info', '/gscratch/scrubbed/srlab/nxf.si5WitkIa9/K43/configs/rna_mode.info']" finished abnormally, OS return value: -6
None

In case you have troubles running SPAdes, you can report an issue on our GitHub repository github.com/ablab/spades
Please provide us with params.txt and spades.log files from the output directory.

SPAdes log can be found here: /gscratch/scrubbed/srlab/nxf.si5WitkIa9/spades.log

Thank you for using rnaSPAdes! If you use it in your research, please cite:

Bushmanova, E., Antipov, D., Lapidus, A. and Prjibelski, A.D., 2019. rnaSPAdes: a de novo transcriptome assembler and its application to RNA-Seq data. GigaScience, 8(9), p.giz100.
doi.org/10.1093/gigascience/giz100

Command error:
INFO: Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred
INFO: Environment variable SINGULARITYENV_NXF_TASK_WORKDIR is set, but APPTAINERENV_NXF_TASK_WORKDIR is preferred
INFO: Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred
/usr/local/share/spades/spades_pipeline/support.py:488: SyntaxWarning: invalid escape sequence '\d'
return [atoi(c) for c in re.split("(\d+)", text)]

Work dir:
/mmfs1/gscratch/scrubbed/strigg/analyses/20241104_denovo/work/45/138d60fae9524f20eb79462e1b8d06

Tip: you can replicate the issue by changing to the process work dir and entering the command bash .command.run

Relevant files

No response

System information

No response

[avani-bhojwani]

Yes, it seems to be a memory issue. I can update the resources for this step