Running into out of memory issues when working with a large number of files.

[karthikram-takara]

Bug report

When a large number of files (>5000) are needed to be moved/copied over using publishDir, nextflow fails with the following error:

error [java.lang.OutOfMemoryError]: unable to create native thread: possibly out of memory or process/resource limits reached
Feb-01 14:38:05.816 [Task monitor] ERROR nextflow.processor.TaskProcessor - Execution aborted due to an unexpected error
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached

From what I can see, the files are created in the workDir, but the failure only happens during the publishDir directive is being enforced.

I have already tried changing different parameters using NXF_OPTS, _JAVA_OPTIONS among others, none of them seem to help fix this issue.

Expected behavior and actual behavior

Expected behavior: output files are copied/moved to the publishDir.
Actual behavior: Pipeline fails with

ERROR ~ Execution aborted due to an unexpected error

 -- Check '.nextflow.log' file for details

with the above detailed error message printed in the log file.

Steps to reproduce the problem

To simulate the scenario, I created the following dummy script and the dummy pipeline that just creates 20,000 files and copies it to another directory using publishDir. I get the same error as above when running it. Dummy script and nextflow pipeline below:

import os

for i in range(20000):
    filename = f"file_{i}.txt"
    with open(filename, "w") as f:
        pass

#!/usr/bin/env nextflow

params.output_dir = ''

process create_files {
        publishDir "${params.output_dir}", mode:'copy'


        output:
                path("*.txt")

        script:
                """
                python /path_to_python_script/test_20000_files/create_20k_files.py
                """
}

workflow {
        create_files()
}

Program output

nextflow.log

Environment

• Nextflow version: 23.10.1.5891
• Java version: openjdk 21-internal 2023-09-19
• Operating system: Linux
• Bash version: GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu)

[bentsherman]

How much CPUs and memory do you have? And what JVM settings have you tried?

[karthikram-takara]

The server this is being run on has 64 CPU threads and 192 GB RAM.

I have tried passing limits starting from "-Xms5g -Xmx24g" to "-Xms20g -Xmx140g" through both NXF_OPTS and _JAVA_OPTIONS alternately, neither of them help complete the pipeline. I get the same error each time.

[bentsherman]

I think you have to use NXF_JVM_ARGS to apply those options correctly. See this blog post for more details.

The default max heap size is usually only 25%, and Nextflow uses a few thread pools which are sized based on the number of CPUs. Still, even with that many CPUs I wouldn't expect Nextflow to use 192/4 = 48 GB of memory...

[pditommaso]

Umm, looking at the stack trace it's using virtual threads and it's failing to create them

java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at java.base/java.lang.Thread.start0(Native Method)
	at java.base/java.lang.Thread.start(Thread.java:1553)
	at java.base/java.lang.System$2.start(System.java:2573)
	at java.base/java.util.concurrent.ThreadPerTaskExecutor.start(ThreadPerTaskExecutor.java:249)
	at java.base/java.util.concurrent.ThreadPerTaskExecutor.submit(ThreadPerTaskExecutor.java:287)
	at java.base/java.util.concurrent.ThreadPerTaskExecutor.submit(ThreadPerTaskExecutor.java:293)
	at nextflow.processor.PublishDir.apply1(PublishDir.groovy:337)
	at nextflow.processor.PublishDir.apply0(PublishDir.groovy:241)
	at nextflow.processor.PublishDir.apply(PublishDir.groovy:299)
	at nextflow.processor.PublishDir$apply.call(Unknown Source)

[bentsherman]

I see you are using 23.10 and Java 21 which enables virtual threads by default. So when the process publishes 20k files it will create 20k virtual threads at once, there is no limit. But virtual threads are lightweight, you should be able to have millions of virtual threads easily

In any case, it's worth trying without virtual threads:

export NXF_ENABLE_VIRTUAL_THREADS=false

[bentsherman]

I just tried your example and had no issues. Nextflow mem usage peaked around 500MB. For what it's worth, I used Java 21 Corretto. Are you on a shared system? Maybe there were other processes / users hogging the memory.

[karthikram-takara]

Thanks @bentsherman, the NXF_ENABLE_VIRTUAL_THREADS setting might be the magic bullet. It works on my toy example and I am now testing it with the actual pipeline which creates > 10,000 files. Will update here once I finish that test.

As for your second question, I am the only user on this server. But the difference might be due to my more stringent ulimit settings (which I am unable to change).

> ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 766321
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1048568
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

[bentsherman]

Our ulimit settings are pretty similar. It's weird, it's as if your JVM is using platform threads under the hood, because ~10,000 platform threads would probably be enough to exhaust your memory. Makes me wonder if some JVMs haven't implemented virtual threads properly.

Seems unlikely. But if you want to try and you have control over your JVM install, maybe try one of our recommended JVM distros (the docs say 17 but use 21).

[karthikram-takara]

Hi @bentsherman and @pditommaso I can confirm that disabling NXF_ENABLE_VIRTUAL_THREADS fixed my issue. Thanks for your help!

[jmuhlich]

I ran into this same thread exhaustion issue on Java 21 with a large number of output files. NXF_ENABLE_VIRTUAL_THREADS=false fixes the problem, as does running under Java 17. I was able to install java 21.0.4-tem with sdkman as @bentsherman requested and the problem still occurs. The platform is Linux - CentOS 7.9.2009. nextflow version is 24.04.4. ulimit -u shows 4096, I was able to watch the nextflow process reach 4000+ threads before dying, and I have about 8500 output files in this case (>> 4096).

[bentsherman]

Folks, we are going to disable virtual threads by default in the next edge release since there are still some scaling problems. It looks like for large runs with many published outputs, we need to add some queue limits which are not provided by virtual threads on their own. See #4995 for further discussion