Skip to content

Solve queueSize exceeded when using job arrays #6047

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

jorgee
Copy link
Contributor

@jorgee jorgee commented May 7, 2025

close #5920

  • Modifying canSubmit to avoid to avoid exceed executor.queueSize parameter.
  • Printing warning when array size exceeds the queueSize. This can make the task array 'unsubmitable' (Not sure if we should abort the execution in this situation)
  • Add unit tests

Tested with this pipeline

process test {
    array 10 
    input:
    val x

    """
    echo $x
    """
}

workflow {
   Channel.of(1..20) | test
}

With a config with awsbatch executor and different executor.queueSize values:

  • size 15: Only one job submitted at a time in AWS Batch.
  • size 5: Prints a warning but run continues without the task array jobs submitted.

@jorgee jorgee linked an issue May 7, 2025 that may be closed by this pull request
Copy link

netlify bot commented May 7, 2025

Deploy Preview for nextflow-docs-staging canceled.

Name Link
🔨 Latest commit 25a6bb3
🔍 Latest deploy log https://app.netlify.com/sites/nextflow-docs-staging/deploys/681b8c25f6dd3e00081a4ac8

@jorgee jorgee requested a review from bentsherman May 8, 2025 13:43
@jorgee jorgee changed the title 5920 job array with queuesize exceeds queuesize Solve queueSize exceeded when using job arrays May 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Job Array with queueSize exceeds queueSize
1 participant