Process outputs as they become available

[magicmaaaaan]

I have a process that outputs a lot of files but can be quite slow between each output. Each output can then be processed independently. Here is a simulation:

process task1 {
    output:
    path "*.txt"

    """
    touch a.txt
    sleep 10
    touch b.txt
    sleep 10
    touch c.txt
    """
}

process task2 {
    input:
    path src

    """
    cat $src
    """
}

workflow {
    task1 | flatten | task2
}

The way this currently works, task1 has to completely finish before task2 can process the outputs in parallel. Is there a way for task2 to start processing outputs as they become available?

I'm not sure if this is even possible since you'd have to know when an output is finished writing but wanted to ask just in case. Something like a.txt is detected but not emitted, b.txt is detected and emits a.txt, c.txt is detected and emits b.txt, and the process finishes and emits c.txt.

[pditommaso]

The output created by task1 are emitted once the tasks complete, and therefore the downstream task can only be trigger once task1 finishes

[magicmaaaaan]

That's how I interpreted the operation as well. Thank you for confirming!