Error in rnaSpades when run from singularity image

[IdoBar]

Description of the bug

rnaSPADES step fails with the following error:

Process `NFCORE_DENOVOTRANSCRIPT:DENOVOTRANSCRIPT:SPADES (pooled_reads)` terminated with an error exit status (12)
....
spades/spades_pipeline/support.py:488: SyntaxWarning: invalid escape sequence '\d'
    return [atoi(c) for c in re.split("(\d+)", text)]

I'm wondering it's a lack of memory or if it's related to this issue that's been fixed in a recent commit, but not in the Singularity image (v4.0.0).

Command used and terminal output

$ nextflow run nf-core/denovotranscript -r 1.0.0 --input Mnova_samplesheet.csv --transrate_reference $WORKDIR/GIU3625_Humpback_whale.transcript.fasta --outdir assembly_results -with-tower -profile apptainer,bunya -c /home/ibar/.nextflow/bunya.config

Error executing process > 'NFCORE_DENOVOTRANSCRIPT:DENOVOTRANSCRIPT:SPADES (pooled_reads)'

Caused by:
  Process `NFCORE_DENOVOTRANSCRIPT:DENOVOTRANSCRIPT:SPADES (pooled_reads)` terminated with an error exit status (12)


Command executed:

  spades.py \
      --rna \
      --threads 12 \
      --memory 72 \
       \
      -1 pooled_reads_1.merged.fastq.gz -2 pooled_reads_2.merged.fastq.gz   \
       \
      -o ./
  mv spades.log pooled_reads.spades.log
  
  if [ -f scaffolds.fasta ]; then
      mv scaffolds.fasta pooled_reads.scaffolds.fa
      gzip -n pooled_reads.scaffolds.fa
  fi
  if [ -f contigs.fasta ]; then
      mv contigs.fasta pooled_reads.contigs.fa
      gzip -n pooled_reads.contigs.fa
  fi
  if [ -f transcripts.fasta ]; then
      mv transcripts.fasta pooled_reads.transcripts.fa
      gzip -n pooled_reads.transcripts.fa
  fi
  if [ -f soft_filtered_transcripts.fasta ]; then
      mv soft_filtered_transcripts.fasta pooled_reads.soft_filtered_transcripts.fa
      gzip -n pooled_reads.soft_filtered_transcripts.fa
  fi
  if [ -f hard_filtered_transcripts.fasta ]; then
      mv hard_filtered_transcripts.fasta pooled_reads.hard_filtered_transcripts.fa
      gzip -n pooled_reads.hard_filtered_transcripts.fa
  fi
  if [ -f assembly_graph_with_scaffolds.gfa ]; then
      mv assembly_graph_with_scaffolds.gfa pooled_reads.assembly.gfa
      gzip -n pooled_reads.assembly.gfa
  fi
  
  if [ -f gene_clusters.fasta ]; then
      mv gene_clusters.fasta pooled_reads.gene_clusters.fa
      gzip -n pooled_reads.gene_clusters.fa
  fi
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_DENOVOTRANSCRIPT:DENOVOTRANSCRIPT:SPADES":
      spades: $(spades.py --version 2>&1 | sed -n 's/^.*SPAdes genome assembler v//p')
  END_VERSIONS

Command exit status:
  12

Command output:
    0:16:08.560  6841M / 7645M INFO    General                 (kmer_splitters.hpp        : 128)   Processed 29316762 reads
    0:16:12.854  6841M / 7645M INFO    General                 (kmer_splitters.hpp        : 128)   Processed 39038979 reads
    0:16:17.374  6841M / 7645M INFO    General                 (kmer_splitters.hpp        : 128)   Processed 48782744 reads
    0:16:21.645  6841M / 7647M INFO    General                 (kmer_splitters.hpp        : 128)   Processed 58498936 reads
    0:16:26.038  6841M / 7647M INFO    General                 (kmer_splitters.hpp        : 128)   Processed 68271535 reads
    0:16:30.401  6841M / 7649M INFO    General                 (kmer_splitters.hpp        : 128)   Processed 78073658 reads
    0:16:35.051  6841M / 7649M INFO    General                 (kmer_splitters.hpp        : 128)   Processed 87867580 reads
    0:16:39.744  6841M / 7649M INFO    General                 (kmer_splitters.hpp        : 128)   Processed 97673676 reads
    0:16:48.487  6841M / 7652M INFO    General                 (kmer_splitters.hpp        : 128)   Processed 107471525 reads
    0:16:53.077  6841M / 7652M INFO    General                 (kmer_splitters.hpp        : 128)   Processed 117319145 reads
    0:17:02.054  6841M / 7652M INFO    General                 (kmer_splitters.hpp        : 128)   Processed 136865777 reads
    0:18:05.624  6841M / 7660M INFO    General                 (kmer_splitters.hpp        : 128)   Processed 273186636 reads
    0:20:13.473  6841M / 7660M INFO    General                 (kmer_splitters.hpp        : 128)   Processed 545964005 reads
    0:24:23.811  6841M / 7662M INFO    General                 (kmer_splitters.hpp        : 128)   Processed 1075403253 reads
    0:32:41.601  6841M / 7671M INFO    General                 (kmer_splitters.hpp        : 128)   Processed 2154882891 reads
    0:37:55.337     1M / 7671M INFO    General                 (kmer_splitters.hpp        : 134)   Used 2804507304 reads
    0:37:55.355     1M / 7671M INFO    General                 (kmer_index_builder.hpp    : 264)   Starting k-mer counting.
    0:37:55.364     1M / 7671M ERROR   General                 (mmapped_reader.hpp        :  52)   mmap(2) failed. Reason: Cannot allocate memory. Error code: 12
  === Stack Trace ===
    0:37:55.364     1M / 7671M ERROR   General                 (mmapped_reader.hpp        :  52)   mmap(2) failed. Reason: Cannot allocate memory. Error code: 12
  === Stack Trace ===
  /usr/local/bin/spades-core(+0x61a97) [0x55605b40ca97]
  /usr/local/bin/spades-core(+0x7cae0) [0x55605b427ae0]
  /usr/local/bin/spades-core(+0x2888cd) [0x55605b6338cd]
  /usr/local/bin/spades-core(+0x28a781) [0x55605b635781]
  /usr/local/bin/../lib/libgomp.so.1(+0x186d9) [0x7f61b79976d9]
  /lib/x86_64-linux-gnu/libc.so.6(+0x89134) [0x7f61b753d134]
  /lib/x86_64-linux-gnu/libc.so.6(__clone+0x40) [0x7f61b75bca40]
  /usr/local/bin/spades-core(+0x61a97) [0x55605b40ca97]
  /usr/local/bin/spades-core(+0x7cae0) [0x55605b427ae0]
  /usr/local/bin/spades-core(+0x2888cd) [0x55605b6338cd]
  /usr/local/bin/spades-core(+0x28a781) [0x55605b635781]
  /usr/local/bin/../lib/libgomp.so.1(+0x186d9) [0x7f61b79976d9]
  /lib/x86_64-linux-gnu/libc.so.6(+0x89134) [0x7f61b753d134]
  /lib/x86_64-linux-gnu/libc.so.6(__clone+0x40) [0x7f61b75bca40]
  
  
  == Error ==  system call for: "['/usr/local/bin/spades-core', 'K33/configs/config.info', 'K33/configs/mda_mode.info', 'K33/configs/rna_mode.info']" finished abnormally, OS return value: 12
  None
  
  In case you have troubles running SPAdes, you can report an issue on our GitHub repository github.com/ablab/spades
  Please provide us with params.txt and spades.log files from the output directory.
  
  SPAdes log can be found here: spades.log
  
  Thank you for using rnaSPAdes! If you use it in your research, please cite:
  
    Bushmanova, E., Antipov, D., Lapidus, A. and Prjibelski, A.D., 2019. rnaSPAdes: a de novo transcriptome assembler and its application to RNA-Seq data. GigaScience, 8(9), p.giz100.
    doi.org/10.1093/gigascience/giz100

Command error:
  /usr/local/share/spades/spades_pipeline/support.py:488: SyntaxWarning: invalid escape sequence '\d'
    return [atoi(c) for c in re.split("(\d+)", text)]

Work dir:
  /scratch/project_mnt/S0016/sandbox/OTE14085/Mnova_denovotranscript_assembly/work/db/51e9b16f8fe073849eefcca37e7af1

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

Relevant files

No response

System information

• Nextflow version (eg. 24.04.4)
• Hardware: HPC
• Executor: slurm
• Container engine: Apptainer
• OS: Rocky Linux release 8.10 (Green Obsidian)
• Version of nf-core/denovotranscript: 1.0.0

[avani-bhojwani]

I suspect it ran out of memory first, and then ran into the SyntaxWarning. I can increase the memory allocated for this step

[IdoBar]

Is there any updated (or older) Spades container image without this issue?

[avani-bhojwani]

I've checked the latest Spades container image. It still has the SyntaxWarning. This is because it is built using Spades release 4.0.0.
The SyntaxWarning has been fixed in development after that - but these new changes are still in a "pre-release".

However, this shouldn't affect the actual assembly (based on what I can see in the code and what another user has mentioned in the issue you've linked). The main issue for this pipeline seems to be that not enough resources were provided. During my testing, this step ran successfully, most likely because I was using a smaller dataset than you. Could you let me know what kind of data you are using - number of samples and approx. millions of reads per sample? This will help me make sure I test on large enough datasets in the future.

[avani-bhojwani]

You can also use a custom config to increase the resources allocation for this step. Please see the nf-core documentation on how to do this here

[IdoBar]

Thanks for the reply, I can make the changes in the config files (I'm pretty familiar with configuring Nextflow pipelines by now...).
I'll try it again and update you later.
Just note that I'm using a more gradual increase of resources on retries with the following configuration:

withLabel:'mc_huge|process_high_memory'{
        cpus = { check_max( 14 + (2 * task.attempt), 'cpus' ) }
        memory = { check_max( 224.GB + (32.GB * task.attempt), 'memory' ) }
        time = { check_max( 80.h + (40.h * task.attempt), 'time' ) }
    }

Cheers, Ido