Standardized "versions" output format and handling

[stevekm]

Outputting the versions of software involved in each Nextflow process has been an important aspect of pipelines for a while now.

A lot of pipelines have been outputting the versions in a manner like this

process FASTQC {

    input:
    tuple val(meta), path(reads)

    output:
    path  "versions.yml", emit: versions

    script:
    """
    cat <<-END_VERSIONS > versions.yml
    "${task.process}":
        fastqc: \$( fastqc --version | sed -e "s/FastQC v//g" )
    END_VERSIONS
    """
}

Where a YAML is created inside each process script block, outputted and collected, and then parsed downstream. Often for the purpose of creating a subsequent YAML file (and/or HTML table) for inclusion with MultiQC.

With the new topic Channels available, this method may be obsolete. Some newer methods for outputting the versions may be seen demo'd here;

https://github.com/stevekm/nextflow-demos/tree/master/topic-channels-versions

here is one example;

process FASTQC {
    tag "${id}"
    container "quay.io/biocontainers/fastqc:0.11.9--0"

    input:
    val(id)

    output:
    tuple val(task.process), val(task.container), val("fastqc"), eval('fastqc --version | sed -e \'s/FastQC v//g\''), topic: versions

    script:
    """
    """
}

This is a pretty nice improvement. However, it leaves open the question of "what should we put in these 'versions' topic channels, and how should we use it?"

I was wondering if the Nextflow dev team & community had any thoughts on this? Is this something that might be helpful to have some sort of standard method?

The method I have been prototyping is like this

// using the output schema shown above
// collect all the versions in a list of Map's
// so that it can be parsed easily with some downstream process
// create one single list of all the unique version map's
channel.topic('versions').map { proc, container, tool, vers ->
        return [
            "process": proc,
            "container": container,
            "software": tool,
            "version": vers
        ]
    }.unique().collect().view()

I am wondering what other methods may have been proposed for accomplishing this, and if some method such as this could be decided upon for inclusion with Nextflow official documentation. And by extension, if some method like this could be standardized, perhaps some extra helpers could be built into Nextflow to make it even easier to collect and parse these versions outputs?

[clintval]

@stevekm, I open sourced this plugin today and am waiting on registry approval:

• https://github.com/fulcrumgenomics/nf-versions

It partially makes the problem you describe easier to manage.

I'd be curious of your thoughts, and PRs welcome!