OOM handling for jobstep plugin

[selten]

Software Versions

$ snakemake --version
9.9.0
$ conda list | grep "snakemake-executor-plugin-slurm"
snakemake-executor-plugin-slurm 1.6.0              pyhdfd78af_0    bioconda
snakemake-executor-plugin-slurm-jobstep 0.3.0              pyhdfd78af_0    bioconda
$ sinfo --version
slurm 24.11.

Describe the bug
When the slurm_jobstep hits OOM, it keeps retrying even though it hit OOM. I would expect that the system would be resubmitted with more RAM.

Logs

~# sacct -j [JOBID]
JobID           JobName  Partition    Account  AllocCPUS      State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
[JOBID]      [JOBNAME]-+      debug    default          1    RUNNING      0:0
[JOBID].bat+      batch               default          1    RUNNING      0:0
[JOBID].0    python3.12               default          1 OUT_OF_ME+    0:125
[JOBID].1    python3.12               default          1 OUT_OF_ME+    0:125
[JOBID].2    python3.12               default          1 OUT_OF_ME+    0:125
[JOBID].3    python3.12               default          1 OUT_OF_ME+    0:125
[JOBID].4    python3.12               default          1 OUT_OF_ME+    0:125
[JOBID].5    python3.12               default          1 OUT_OF_ME+    0:125
[JOBID].6    python3.12               default          1 OUT_OF_ME+    0:125
[JOBID].7    python3.12               default          1 OUT_OF_ME+    0:125
[JOBID].8    python3.12               default          1 OUT_OF_ME+    0:125
[JOBID].9    python3.12               default          1 OUT_OF_ME+    0:125

Minimal example

from snakemake.shell import shell

rule use_ram:
    input:
      input = '/dev/zero'
    threads: 1
    resources:
      cores=1,
      mem_mb=10240,
      partition='debug',
      nodes=1
    retries: 10
    shell:
        "head -c 12G    {input} | tail"

Additional context
What I would expect is that given it runs into OOM, it should just cancel the sbatch or return to the starting snakemake job to potentially get more RAM. I am well aware that in this specific example that would not happen, but I still think that should be the behaviour instead of retrying consistently while it hit OOM.

[cmeesters]

Hi,

thanks for reporting your issue. I am not sure, why you think this is related to the jobstep executor or Snakemake at all - nor how you managed the several jobsteps with the example snippet. Essentially, every failure causes the workflow to halt further submissions, if not instructed otherwise. It then keeps running until the last still running job finishes.

Also, there is no import statement needed for your minimal example.

When designing a workflow, you can refer to the dynamic resource section (look for “attempt”) to achieve your desired behaviour. Please note, that submitting with more RAM is case specific, as different tools show different RAM usage. You can also make the RAM specification depended on input size, to avoid a crash in the first place.

If you are using a 3rd party workflow, I hope that the curators value your input.

Do you still think you hit a bug? Then please give a verbose log and indicate your command line and configuration settings. Also, please describe why it is a bug. Perhaps give a link or code of your Snakefile, too.

[selten]

I believe my description was quite poor. Let me try again.

Snakefile:

APPTAINER_CMD="/PATH/TO/apptainer exec /PATH/TO/ub_2004.sif"

rule use_ram:
    input:
      input = '/dev/zero'
    resources:
      cpus_per_task=1,
      mem_mb = lambda wildcards, attempt: attempt * 1000,
      partition='debug',
      nodes=1
    retries: 13
    shell:
        "{APPTAINER_CMD} head -c 12G    {input} | tail"

Main Snakemake log:

cat .snakemake/log/2025-09-16T141542.121851.snakemake.log
None
host: NODENAME
Building DAG of jobs...
SLURM run ID: 8c74f2d9-1138-42de-8b92-d225976e013b
Using shell: /usr/bin/bash
Provided remote nodes: 1
Job stats:
job        count
-------  -------
use_ram        1
total          1

Select jobs to execute...
Execute 1 jobs...
[Tue Sep 16 14:15:42 2025]
rule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=1000, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
No wall time information given. This might or might not work on your cluster. If not, specify the resource runtime in your rule or as a reasonable default via --default-resources.
Job 0 has been submitted with SLURM jobid 7405113 (log: /PATH/TO/snakemake/.snakemake/slurm_logs/rule_use_ram/7405113.log).
[Tue Sep 16 14:16:22 2025]
Error in rule use_ram:
    message: SLURM-job '7405113' failed, SLURM status is: 'FAILED'. For further error details see the cluster/cloud log and the log files of the involved rule(s).
    jobid: 0
    input: /dev/zero
    log: /PATH/TO/snakemake/.snakemake/slurm_logs/rule_use_ram/7405113.log (check log file(s) for error details)
    shell:
        /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
        (command exited with non-zero exit code)
    external_jobid: 7405113
Trying to restart job 0.
Select jobs to execute...
Execute 1 jobs...
[Tue Sep 16 14:16:22 2025]
rule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=2000, mem_mib=1908, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
No wall time information given. This might or might not work on your cluster. If not, specify the resource runtime in your rule or as a reasonable default via --default-resources.
Job 0 has been submitted with SLURM jobid 7405273 (log: /PATH/TO/snakemake/.snakemake/slurm_logs/rule_use_ram/7405273.log).
[Tue Sep 16 14:17:12 2025]
Error in rule use_ram:
    message: SLURM-job '7405273' failed, SLURM status is: 'FAILED'. For further error details see the cluster/cloud log and the log files of the involved rule(s).
    jobid: 0
    input: /dev/zero
    log: /PATH/TO/snakemake/.snakemake/slurm_logs/rule_use_ram/7405273.log (check log file(s) for error details)
    shell:
        /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
        (command exited with non-zero exit code)
    external_jobid: 7405273
Trying to restart job 0.
Select jobs to execute...
Execute 1 jobs...
[Tue Sep 16 14:17:12 2025]
rule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=3000, mem_mib=2862, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
No wall time information given. This might or might not work on your cluster. If not, specify the resource runtime in your rule or as a reasonable default via --default-resources.
Job 0 has been submitted with SLURM jobid 7405450 (log: /PATH/TO/snakemake/.snakemake/slurm_logs/rule_use_ram/7405450.log).
[Tue Sep 16 14:18:42 2025]
Error in rule use_ram:
    message: SLURM-job '7405450' failed, SLURM status is: 'FAILED'. For further error details see the cluster/cloud log and the log files of the involved rule(s).
    jobid: 0
    input: /dev/zero
    log: /PATH/TO/snakemake/.snakemake/slurm_logs/rule_use_ram/7405450.log (check log file(s) for error details)
    shell:
        /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
        (command exited with non-zero exit code)
    external_jobid: 7405450
Trying to restart job 0.
Select jobs to execute...
Execute 1 jobs...
[Tue Sep 16 14:18:42 2025]
rule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=4000, mem_mib=3815, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
No wall time information given. This might or might not work on your cluster. If not, specify the resource runtime in your rule or as a reasonable default via --default-resources.
Job 0 has been submitted with SLURM jobid 7405668 (log: /PATH/TO/snakemake/.snakemake/slurm_logs/rule_use_ram/7405668.log).
[Tue Sep 16 14:20:52 2025]
Error in rule use_ram:
    message: SLURM-job '7405668' failed, SLURM status is: 'FAILED'. For further error details see the cluster/cloud log and the log files of the involved rule(s).
    jobid: 0
    input: /dev/zero
    log: /PATH/TO/snakemake/.snakemake/slurm_logs/rule_use_ram/7405668.log (check log file(s) for error details)
    shell:
        /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
        (command exited with non-zero exit code)
    external_jobid: 7405668
Trying to restart job 0.
Select jobs to execute...
Execute 1 jobs...
[Tue Sep 16 14:20:52 2025]
rule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=5000, mem_mib=4769, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
No wall time information given. This might or might not work on your cluster. If not, specify the resource runtime in your rule or as a reasonable default via --default-resources.
Job 0 has been submitted with SLURM jobid 7406077 (log: /PATH/TO/snakemake/.snakemake/slurm_logs/rule_use_ram/7406077.log).
[Tue Sep 16 14:22:12 2025]
Error in rule use_ram:
    message: SLURM-job '7406077' failed, SLURM status is: 'FAILED'. For further error details see the cluster/cloud log and the log files of the involved rule(s).
    jobid: 0
    input: /dev/zero
    log: /PATH/TO/snakemake/.snakemake/slurm_logs/rule_use_ram/7406077.log (check log file(s) for error details)
    shell:
        /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
        (command exited with non-zero exit code)
    external_jobid: 7406077
Trying to restart job 0.
Select jobs to execute...
Execute 1 jobs...
[Tue Sep 16 14:22:12 2025]
rule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=6000, mem_mib=5723, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
No wall time information given. This might or might not work on your cluster. If not, specify the resource runtime in your rule or as a reasonable default via --default-resources.
Job 0 has been submitted with SLURM jobid 7406437 (log: /PATH/TO/snakemake/.snakemake/slurm_logs/rule_use_ram/7406437.log).
Terminating processes on user request, this might take some time.

All looks fine up to here.

.snakemake/slurm_logs/rule_use_ram/7405113.log:

host: NODENAME
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided remote nodes: 1
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:15:43 2025]
rule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=1000, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
host: NODENAME
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:15:44 2025]
localrule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=1000, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=/tmp, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
INFO:    fuse2fs not found, will not be able to mount EXT3 filesystems
INFO:    gocryptfs not found, will not be able to use gocryptfs
slurmstepd: error: Detected 2 oom_kill events in StepId=7405113.0. Some of the step tasks have been OOM Killed.
srun: error: NODENAME: task 0: Out Of Memory
[Tue Sep 16 14:15:46 2025]
Error in rule use_ram:
    message: None
    jobid: 0
    input: /dev/zero
    shell:
        /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
        (command exited with non-zero exit code)
Trying to restart job 0.
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:15:46 2025]
rule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=2000, mem_mib=1908, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
host: NODENAME
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:15:46 2025]
localrule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=2000, mem_mib=1908, disk_mb=1000, disk_mib=954, tmpdir=/tmp, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
INFO:    fuse2fs not found, will not be able to mount EXT3 filesystems
INFO:    gocryptfs not found, will not be able to use gocryptfs
slurmstepd: error: Detected 7 oom_kill events in StepId=7405113.1. Some of the step tasks have been OOM Killed.
srun: error: NODENAME: task 0: Out Of Memory
[Tue Sep 16 14:15:48 2025]
Error in rule use_ram:
    message: None
    jobid: 0
    input: /dev/zero
    shell:
        /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
        (command exited with non-zero exit code)
Trying to restart job 0.
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:15:48 2025]
rule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=3000, mem_mib=2862, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
host: NODENAME
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:15:49 2025]
localrule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=3000, mem_mib=2862, disk_mb=1000, disk_mib=954, tmpdir=/tmp, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
INFO:    fuse2fs not found, will not be able to mount EXT3 filesystems
INFO:    gocryptfs not found, will not be able to use gocryptfs
slurmstepd: error: Detected 2 oom_kill events in StepId=7405113.2. Some of the step tasks have been OOM Killed.
srun: error: NODENAME: task 0: Out Of Memory
[Tue Sep 16 14:15:51 2025]
Error in rule use_ram:
    message: None
    jobid: 0
    input: /dev/zero
    shell:
        /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
        (command exited with non-zero exit code)
Trying to restart job 0.
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:15:51 2025]
rule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=4000, mem_mib=3815, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
host: NODENAME
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:15:51 2025]
localrule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=4000, mem_mib=3815, disk_mb=1000, disk_mib=954, tmpdir=/tmp, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
INFO:    fuse2fs not found, will not be able to mount EXT3 filesystems
INFO:    gocryptfs not found, will not be able to use gocryptfs
slurmstepd: error: Detected 2 oom_kill events in StepId=7405113.3. Some of the step tasks have been OOM Killed.
srun: error: NODENAME: task 0: Out Of Memory
[Tue Sep 16 14:15:53 2025]
Error in rule use_ram:
    message: None
    jobid: 0
    input: /dev/zero
    shell:
        /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
        (command exited with non-zero exit code)
Trying to restart job 0.
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:15:53 2025]
rule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=5000, mem_mib=4769, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
host: NODENAME
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:15:54 2025]
localrule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=5000, mem_mib=4769, disk_mb=1000, disk_mib=954, tmpdir=/tmp, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
INFO:    fuse2fs not found, will not be able to mount EXT3 filesystems
INFO:    gocryptfs not found, will not be able to use gocryptfs
slurmstepd: error: Detected 2 oom_kill events in StepId=7405113.4. Some of the step tasks have been OOM Killed.
srun: error: NODENAME: task 0: Out Of Memory
[Tue Sep 16 14:15:55 2025]
Error in rule use_ram:
    message: None
    jobid: 0
    input: /dev/zero
    shell:
        /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
        (command exited with non-zero exit code)
Trying to restart job 0.
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:15:55 2025]
rule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=6000, mem_mib=5723, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
host: NODENAME
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:15:56 2025]
localrule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=6000, mem_mib=5723, disk_mb=1000, disk_mib=954, tmpdir=/tmp, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
INFO:    fuse2fs not found, will not be able to mount EXT3 filesystems
INFO:    gocryptfs not found, will not be able to use gocryptfs
slurmstepd: error: Detected 2 oom_kill events in StepId=7405113.5. Some of the step tasks have been OOM Killed.
srun: error: NODENAME: task 0: Out Of Memory
[Tue Sep 16 14:15:58 2025]
Error in rule use_ram:
    message: None
    jobid: 0
    input: /dev/zero
    shell:
        /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
        (command exited with non-zero exit code)
Trying to restart job 0.
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:15:58 2025]
rule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=7000, mem_mib=6676, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
host: NODENAME
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:15:59 2025]
localrule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=7000, mem_mib=6676, disk_mb=1000, disk_mib=954, tmpdir=/tmp, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
INFO:    fuse2fs not found, will not be able to mount EXT3 filesystems
INFO:    gocryptfs not found, will not be able to use gocryptfs
slurmstepd: error: Detected 7 oom_kill events in StepId=7405113.6. Some of the step tasks have been OOM Killed.
srun: error: NODENAME: task 0: Out Of Memory
[Tue Sep 16 14:16:00 2025]
Error in rule use_ram:
    message: None
    jobid: 0
    input: /dev/zero
    shell:
        /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
        (command exited with non-zero exit code)
Trying to restart job 0.
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:16:00 2025]
rule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=8000, mem_mib=7630, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
host: NODENAME
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:16:01 2025]
localrule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=8000, mem_mib=7630, disk_mb=1000, disk_mib=954, tmpdir=/tmp, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
INFO:    fuse2fs not found, will not be able to mount EXT3 filesystems
INFO:    gocryptfs not found, will not be able to use gocryptfs
slurmstepd: error: Detected 2 oom_kill events in StepId=7405113.7. Some of the step tasks have been OOM Killed.
srun: error: NODENAME: task 0: Out Of Memory
[Tue Sep 16 14:16:03 2025]
Error in rule use_ram:
    message: None
    jobid: 0
    input: /dev/zero
    shell:
        /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
        (command exited with non-zero exit code)
Trying to restart job 0.
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:16:03 2025]
rule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=9000, mem_mib=8584, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
host: NODENAME
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:16:03 2025]
localrule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=9000, mem_mib=8584, disk_mb=1000, disk_mib=954, tmpdir=/tmp, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
INFO:    fuse2fs not found, will not be able to mount EXT3 filesystems
INFO:    gocryptfs not found, will not be able to use gocryptfs
slurmstepd: error: Detected 2 oom_kill events in StepId=7405113.8. Some of the step tasks have been OOM Killed.
srun: error: NODENAME: task 0: Out Of Memory
[Tue Sep 16 14:16:05 2025]
Error in rule use_ram:
    message: None
    jobid: 0
    input: /dev/zero
    shell:
        /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
        (command exited with non-zero exit code)
Trying to restart job 0.
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:16:05 2025]
rule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=10000, mem_mib=9537, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
host: NODENAME
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:16:06 2025]
localrule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=10000, mem_mib=9537, disk_mb=1000, disk_mib=954, tmpdir=/tmp, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
INFO:    fuse2fs not found, will not be able to mount EXT3 filesystems
INFO:    gocryptfs not found, will not be able to use gocryptfs
slurmstepd: error: Detected 7 oom_kill events in StepId=7405113.9. Some of the step tasks have been OOM Killed.
srun: error: NODENAME: task 0: Out Of Memory
[Tue Sep 16 14:16:08 2025]
Error in rule use_ram:
    message: None
    jobid: 0
    input: /dev/zero
    shell:
        /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
        (command exited with non-zero exit code)
Trying to restart job 0.
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:16:08 2025]
rule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=11000, mem_mib=10491, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
host: NODENAME
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:16:08 2025]
localrule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=11000, mem_mib=10491, disk_mb=1000, disk_mib=954, tmpdir=/tmp, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
INFO:    fuse2fs not found, will not be able to mount EXT3 filesystems
INFO:    gocryptfs not found, will not be able to use gocryptfs
slurmstepd: error: Detected 7 oom_kill events in StepId=7405113.10. Some of the step tasks have been OOM Killed.
srun: error: NODENAME: task 0: Out Of Memory
[Tue Sep 16 14:16:10 2025]
Error in rule use_ram:
    message: None
    jobid: 0
    input: /dev/zero
    shell:
        /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
        (command exited with non-zero exit code)
Trying to restart job 0.
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:16:10 2025]
rule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=12000, mem_mib=11445, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
host: NODENAME
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:16:11 2025]
localrule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=12000, mem_mib=11445, disk_mb=1000, disk_mib=954, tmpdir=/tmp, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
INFO:    fuse2fs not found, will not be able to mount EXT3 filesystems
INFO:    gocryptfs not found, will not be able to use gocryptfs
slurmstepd: error: Detected 2 oom_kill events in StepId=7405113.11. Some of the step tasks have been OOM Killed.
srun: error: NODENAME: task 0: Out Of Memory
[Tue Sep 16 14:16:13 2025]
Error in rule use_ram:
    message: None
    jobid: 0
    input: /dev/zero
    shell:
        /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
        (command exited with non-zero exit code)
Trying to restart job 0.
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:16:13 2025]
rule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=13000, mem_mib=12398, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
host: NODENAME
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:16:13 2025]
localrule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=13000, mem_mib=12398, disk_mb=1000, disk_mib=954, tmpdir=/tmp, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
INFO:    fuse2fs not found, will not be able to mount EXT3 filesystems
INFO:    gocryptfs not found, will not be able to use gocryptfs
slurmstepd: error: Detected 2 oom_kill events in StepId=7405113.12. Some of the step tasks have been OOM Killed.
srun: error: NODENAME: task 0: Out Of Memory
[Tue Sep 16 14:16:15 2025]
Error in rule use_ram:
    message: None
    jobid: 0
    input: /dev/zero
    shell:
        /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
        (command exited with non-zero exit code)
Trying to restart job 0.
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:16:15 2025]
rule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=14000, mem_mib=13352, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
host: NODENAME
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Select jobs to execute...
Execute 1 jobs...

[Tue Sep 16 14:16:16 2025]
localrule use_ram:
    input: /dev/zero
    jobid: 0
    reason: Rules with a run or shell declaration but no output are always executed.
    resources: mem_mb=14000, mem_mib=13352, disk_mb=1000, disk_mib=954, tmpdir=/tmp, slurm_account=default, slurm_partition=debug, cpus_per_task=1, partition=debug, nodes=1
Shell command: /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
INFO:    fuse2fs not found, will not be able to mount EXT3 filesystems
INFO:    gocryptfs not found, will not be able to use gocryptfs
slurmstepd: error: Detected 7 oom_kill events in StepId=7405113.13. Some of the step tasks have been OOM Killed.
srun: error: NODENAME: task 0: Out Of Memory
[Tue Sep 16 14:16:18 2025]
Error in rule use_ram:
    message: None
    jobid: 0
    input: /dev/zero
    shell:
        /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
        (command exited with non-zero exit code)
Shutting down, this might take some time.
Exiting because a job execution failed. Look below for error messages
[Tue Sep 16 14:16:18 2025]
Error in rule use_ram:
    message: None
    jobid: 0
    input: /dev/zero
    shell:
        /PATH/TO/apptainer exec /PATH/TO/ub_2004.sif head -c 12G    /dev/zero | tail
        (command exited with non-zero exit code)
Storing output in storage.
WorkflowError:
At least one job did not complete successfully.

So the way I see it, what happens is:
We sbatch with 1000MB RAM, then we srun with 1000MB RAM. Subsequently, it retries the srun with up to 14000MB RAM.
Then once this fails, it will sbatch with 2000MB RAM, then we srun with 1000MB RAM. Subsequently it will retry up to 14000MB RAM.

This keeps on going until it can run the tool. Thus with this configuration it would rerun 91 times to be successful instead of the configured 13 times.

To me, these additional srun's beyond the first OOM are irrelevant. The cgroup will limit to the sbatch value first. The higher srun value should never occur from my viewpoint. Ideally, it would detect that the srun hit OOM limit and exit the sbatch to then resubmit the sbatch with more RAM.

[jd3234]

Hi @cmeesters ,
is there any update on this? Thanks! :)

[cmeesters]

A fix is in preparation, hopefully by the end of the week. Autumn holidays and other non-family obligations kept me busy.