Include metadata in the output of file processing operators

[Gullumluvl]

New feature

Hi, thank you for developing Nextflow!

I am wondering, I build lots of my workflow steps using Nextflow builtin operators. One thing that could be helpful is a feature to keep track of which file was input to splitCsv or countLines operator.

Usage scenario

More precisely, by metadata I mostly mean the source file. This would allow to compute something on each file of a channel, but keep the provenance, so as to easily merge back this information.

For example, let's say we need to know the number of lines of each file (for use as size in groupKey for example):

Each item of the desired output channel should be like [ source_file_path, number_of_lines ].

For splitCsv outputs, having the source file automatically added means we can process lines by source file, and/or use collectFile to gather back lines into files corresponding to their original source.

Suggested implementation

The most basic idea would be to add a specific boolean option, such as withSourceFile:

Channel.fromPath('{a,b,c}.txt')
| countLines(withSourceFile: true)
| view

[a.txt, 5]
[b.txt, 12]
[c.txt, 23]

and similarly for e.g. splitCsv, splitText and others.

Alternatively, one rather transparent but more flexible implementation would be to allow all tuple elements to be passed along, instead of discarding them, so that we can do:

Channel.fromPath('{a,b,c}.txt')
| map { [it.name, it] }
| countLines(elem: 1, passAllElems: true)
| view

["a.txt", 5]
["b.txt", 12]
["c.txt", 23]

Current workaround

Currently, to count lines and keep the source file name, I have to either:

• read the file with groovy code inside a map operator:

  int countlines(fpath) {
      int n = 0
      fpath.eachLine { n++ }
  }
  
  Channel.fromPath('{a,b,c}.txt')
  | map { [ it.name, countlines(it) ] }

• define a process doing the same.

For split* operators such as splitCsv, the best seems to first use a process to insert the filename into the file itself, for example:

process insert_sourcefilename {
    input:
    path(tsv)

    output:
    path('new.tsv')

    """
    sed 's/^/${tsv.name}\\t/' $tsv > new.tsv
    """
}

I hope that's a reasonable suggestion... I am happy to explain more the use cases.

[bentsherman]

Operators like splitCsv and collectFile are convenient for simple use cases, but for more complex uses like a nested scatter-gather it is better to use flatMap and groupTuple.

The split* and count* operators are also available as functions, and each operator is essentially a shorthand for using flatMap with the corresponding function, for example:

Channel.fromPath('*.csv')
  .flatMap { csv -> csv.splitCsv() }

So if you wanted to preserve things like the source file and number of lines, you can do it all in a flatMap:

Channel.fromPath('*.csv')
  .flatMap { csv ->
    def records = csv.splitCsv()
    records.collect { record ->
      tuple( csv, records.size(), record )
    }
  }

There is a similar analogy for collectFile and groupTuple -- if you find yourself stretching the capabilities of collectFile, you might want to use groupTuple instead. You can see an example of this pattern here.

[Gullumluvl]

So if you wanted to preserve things like the source file and number of lines, you can do it all in a flatMap:

Channel.fromPath('*.csv')
  .flatMap { csv ->
    def records = csv.splitCsv()
    records.collect { record ->
      tuple( csv, records.size(), record )
    }
  }

Ah, this is neat! I was not aware of these as functions, my bad! I agree there is no need for more then.