How to specify a scratch output directory in Nextflow?

Question

I have started to read Nexflow's documentation and found that one can specify a scratch directory for the execution. Once the task is complete, one can use the stageOutMode directive to copy the output files from scratch to storeDir.

The output files to be copied are specified by the output directive. My question is the following: is it possible to specify entire directories as output so that they would be copied recursively from scratch to storeDir? If so, how?

Answer 1

By default, the path output qualifier will capture process outputs (files, directories, etc) recursively. All you need to do is specify the (top-level) directory in your output declaration, like in the example below:

nextflow.enable.dsl=2

process test {

    scratch '/tmp/my/path'

    stageOutMode 'copy'

    storeDir '/store/results'

    input:
    val myint

    output:
    path "outdir-${myint}"

    script:
    def outdir = "outdir-${myint}/foo/bar/baz"

    """
    mkdir -p "${outdir}" 
    touch "${outdir}/${myint}.txt"
    """
}

workflow {

    ch = Channel.of( 1..3 )

    test(ch)
}

Setting the stageOutMode directive just changes the how the output files are staged out from the scratch directory to the work directory. I.e. this directive does not change how process results are staged into the storeDir directory.

The storeDir directive changes what finally happens to the files listed in the output declaration such that they are moved from the work directory into the specified storeDir directory.