Solve queueSize exceeded when using job arrays

[jorgee]

close #5920

• Modifying canSubmit to avoid to avoid exceed executor.queueSize parameter.
• Printing warning when array size exceeds the queueSize. This can make the task array 'unsubmitable' (Not sure if we should abort the execution in this situation)
• Add unit tests

Tested with this pipeline

process test {
    array 10 
    input:
    val x

    """
    echo $x
    """
}

workflow {
   Channel.of(1..20) | test
}

With a config with awsbatch executor and different executor.queueSize values:

• size 15: Only one job submitted at a time in AWS Batch.
• size 5: Prints a warning but run continues without the task array jobs submitted.

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	a944cc8
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs-staging/deploys/68387c46a957710008a2059a

[bentsherman]

• change the warning to an error, since the run will hang anyway
• make process maxForks aware of job arrays

[pditommaso]

Should not be task.processor.forksCount * getForksCount(), I mean multiplied?

[bentsherman]

why would it be multiplied?

[pditommaso]

Oh, I was totally confused by the getForksCount naming. This new attribute should be named differently because it overlaps with processor.forksCount that has a complete different meaning. Something like getRunsCount and getArraySize would be much better

[bentsherman]

how are they different in meaning?

[pditommaso]

Oh, I was totally confused by the getForksCount naming. This new attribute should be named differently because it overlaps with processor.forksCount that has a complete different meaning. Something like getRunsCount and getArraySize would be much better

[pditommaso]

Because processor.forksCount is related to maxForks directive, instead here the aim is to controller the max numbers of task runs

[bentsherman]

So "forks" here just refers to the number of concurrent tasks, therefore I see no reason to call it something else in the task handler