Workflow with InitialWorkdirRequirement causes output to be symlinked from /tmp

[hcraT]

Expected Behavior

The workflow posted below produces the following tree structure:

out_dir/
  step1.txt

step1.txt is a normal file.

Actual Behavior

step1.txt is a symbolic link to the /tmp folder where results are computed.

Workflow Code

cwlVersion: v1.1
class: Workflow

requirements:
  SubworkflowFeatureRequirement: {}
  InlineJavascriptRequirement: {}
  StepInputExpressionRequirement: {}

inputs: []

outputs:
  mmm:
    type: Directory
    outputSource: merge_output/output

steps:
  step1:
    in: []
    out: [output]
    run:
      class: CommandLineTool
      baseCommand: [echo]
      stdout: "step1.txt"
      inputs: []
      outputs:
        output:
          type: stdout
      arguments:
        - valueFrom: "step1"
  merge_output:
    in:
      in1: step1/output
    out: [output]
    run:
      requirements:
        InitialWorkDirRequirement:
          listing: |
            ${
              return [
                {
                  class: "Directory",
                  basename: "out_dir",
                  listing: [ inputs.in1 ]
                }
              ];
            }
      class: CommandLineTool
      baseCommand: ["true"]
      inputs:
        in1:
          type: File
      outputs:
        output:
          type: Directory
          outputBinding:
            glob: "out_dir"

If I change the merge_output step to:

  merge_output:
    in:
      in1: step1/output
    out: [output]
    run:
      class: ExpressionTool
      inputs:
        in1: File
      outputs:
        output: Directory
      expression: |
        ${
          return {
            output: {
              class: "Directory",
              basename: "out_dir",
              listing: [ inputs.in1 ]
            }
          };
        }

the output is correct.

Full Traceback

No exception happens but I thought that some useful information
could be part of the log.

INFO /shares/CIBIO-Storage/CO/scratch/sharedCO/tools/executables/cwltool/current/installed/bin/__cwltool_orig__ 3.0
INFO Resolved 'bug_report.cwl' to 'file:///shares/CIBIO-Storage/CO/elaborazioni/sharedCO/dev/pipeline/cwl_my_examples/bug_report.cwl'
DEBUG Parsed job order from command line: {
    "id": "bug_report.cwl"
}
DEBUG [workflow ] initialized from file:///shares/CIBIO-Storage/CO/elaborazioni/sharedCO/dev/pipeline/cwl_my_examples/bug_report.cwl
INFO [workflow ] start
DEBUG [workflow ] inputs {}
DEBUG [workflow ] job step file:///shares/CIBIO-Storage/CO/elaborazioni/sharedCO/dev/pipeline/cwl_my_examples/bug_report.cwl#merge_output not ready
INFO [workflow ] starting step step1
DEBUG [step step1] job input {}
DEBUG [step step1] evaluated job input to {}
INFO [step step1] start
DEBUG [job step1] initializing from _:fe4994ec-fb74-49d8-862c-c14b000c0329 as part of step step1
DEBUG [job step1] {}
DEBUG [job step1] path mappings is {}
DEBUG [job step1] command line bindings is [
    {
        "position": [
            -1000000,
            0
        ],
        "datum": "echo"
    },
    {
        "valueFrom": "step1",
        "position": [
            0,
            0
        ]
    }
]
DEBUG [job step1] initial work dir {}
INFO [job step1] /tmp/c6k2_i_q$ echo \
    step1 > /tmp/c6k2_i_q/step1.txt
DEBUG Could not collect memory usage, job ended before monitoring began.
INFO [job step1] completed success
DEBUG [job step1] outputs {
    "output": {
        "nameroot": "step1",
        "location": "file:///tmp/c6k2_i_q/step1.txt",
        "class": "File",
        "nameext": ".txt",
        "basename": "step1.txt",
        "checksum": "sha1$a7420b6a7186a08cc642bbae5849e869e69b01a0",
        "http://commonwl.org/cwltool#generation": 0,
        "size": 6
    }
}
DEBUG [step step1] produced output {
    "file:///shares/CIBIO-Storage/CO/elaborazioni/sharedCO/dev/pipeline/cwl_my_examples/bug_report.cwl#step1/output": {
        "nameroot": "step1",
        "location": "file:///tmp/c6k2_i_q/step1.txt",
        "class": "File",
        "nameext": ".txt",
        "basename": "step1.txt",
        "checksum": "sha1$a7420b6a7186a08cc642bbae5849e869e69b01a0",
        "http://commonwl.org/cwltool#generation": 0,
        "size": 6
    }
}
INFO [step step1] completed success
DEBUG [job step1] Removing input staging directory /tmp/tmp9bb1kanc
DEBUG [job step1] Removing temporary directory /tmp/tmpzdwiyloo
INFO [workflow ] starting step merge_output
DEBUG [step merge_output] job input {
    "file:///shares/CIBIO-Storage/CO/elaborazioni/sharedCO/dev/pipeline/cwl_my_examples/bug_report.cwl#merge_output/in1": {
        "nameroot": "step1",
        "location": "file:///tmp/c6k2_i_q/step1.txt",
        "class": "File",
        "nameext": ".txt",
        "basename": "step1.txt",
        "checksum": "sha1$a7420b6a7186a08cc642bbae5849e869e69b01a0",
        "http://commonwl.org/cwltool#generation": 0,
        "size": 6
    }
}
DEBUG [step merge_output] evaluated job input to {
    "file:///shares/CIBIO-Storage/CO/elaborazioni/sharedCO/dev/pipeline/cwl_my_examples/bug_report.cwl#merge_output/in1": {
        "nameroot": "step1",
        "location": "file:///tmp/c6k2_i_q/step1.txt",
        "class": "File",
        "nameext": ".txt",
        "basename": "step1.txt",
        "checksum": "sha1$a7420b6a7186a08cc642bbae5849e869e69b01a0",
        "http://commonwl.org/cwltool#generation": 0,
        "size": 6
    }
}
INFO [step merge_output] start
DEBUG [job merge_output] initializing from _:e9912939-fd4f-47ab-a89a-cee5aa851c97 as part of step merge_output
DEBUG [job merge_output] {
    "in1": {
        "nameroot": "step1",
        "location": "file:///tmp/c6k2_i_q/step1.txt",
        "class": "File",
        "nameext": ".txt",
        "basename": "step1.txt",
        "checksum": "sha1$a7420b6a7186a08cc642bbae5849e869e69b01a0",
        "http://commonwl.org/cwltool#generation": 0,
        "size": 6
    }
}
DEBUG [job merge_output] path mappings is {
    "file:///tmp/c6k2_i_q/step1.txt": [
        "/tmp/c6k2_i_q/step1.txt",
        "/tmp/lgx56dfb/out_dir/step1.txt",
        "File",
        false
    ]
}
DEBUG [job merge_output] command line bindings is [
    {
        "position": [
            -1000000,
            0
        ],
        "datum": "true"
    }
]
DEBUG [job merge_output] initial work dir {
    "file:///tmp/c6k2_i_q/step1.txt": [
        "/tmp/c6k2_i_q/step1.txt",
        "/tmp/lgx56dfb/out_dir/step1.txt",
        "File",
        true
    ],
    "_:4801bb3d-f40f-4e2c-b227-57e6b475b608": [
        "_:4801bb3d-f40f-4e2c-b227-57e6b475b608",
        "/tmp/lgx56dfb/out_dir",
        "Directory",
        true
    ]
}
INFO [job merge_output] /tmp/lgx56dfb$ true
DEBUG Could not collect memory usage, job ended before monitoring began.
INFO [job merge_output] completed success
DEBUG [job merge_output] outputs {
    "output": {
        "basename": "out_dir",
        "nameroot": "out_dir",
        "location": "file:///tmp/lgx56dfb/out_dir",
        "class": "Directory",
        "nameext": ""
    }
}
DEBUG [step merge_output] produced output {
    "file:///shares/CIBIO-Storage/CO/elaborazioni/sharedCO/dev/pipeline/cwl_my_examples/bug_report.cwl#merge_output/output": {
        "basename": "out_dir",
        "nameroot": "out_dir",
        "location": "file:///tmp/lgx56dfb/out_dir",
        "class": "Directory",
        "nameext": ""
    }
}
INFO [step merge_output] completed success
INFO [workflow ] completed success
DEBUG [workflow ] outputs {
    "mmm": {
        "basename": "out_dir",
        "nameroot": "out_dir",
        "location": "file:///tmp/lgx56dfb/out_dir",
        "class": "Directory",
        "nameext": ""
    }
}
DEBUG [job merge_output] Removing input staging directory /tmp/tmpfgnuvfpe
DEBUG [job merge_output] Removing temporary directory /tmp/tmps95hvmqr
DEBUG Moving /tmp/lgx56dfb/out_dir to /shares/CIBIO-Storage/CO/elaborazioni/sharedCO/dev/pipeline/cwl_my_examples/out_dir
{
    "mmm": {
        "basename": "out_dir",
        "path": "/shares/CIBIO-Storage/CO/elaborazioni/sharedCO/dev/pipeline/cwl_my_examples/out_dir",
        "location": "file:///shares/CIBIO-Storage/CO/elaborazioni/sharedCO/dev/pipeline/cwl_my_examples/out_dir",
        "class": "Directory",
        "listing": [
            {
                "path": "/shares/CIBIO-Storage/CO/elaborazioni/sharedCO/dev/pipeline/cwl_my_examples/out_dir/step1.txt",
                "location": "file:///shares/CIBIO-Storage/CO/elaborazioni/sharedCO/dev/pipeline/cwl_my_examples/out_dir/step1.txt",
                "class": "File",
                "basename": "step1.txt",
                "checksum": "sha1$a7420b6a7186a08cc642bbae5849e869e69b01a0",
                "size": 6
            }
        ]
    }
}
INFO Final process status is success

Your Environment

I'm using cwltool version 3.0.20201026152241

[mr-c]

@tetron suggests that the directory staging function isn't checking that all the contents are also staged correctly

[tetron]

I think it just moves the directory without considering that files within the directory that are links might need to be copied.

[tetron]

Actually, what should happen is that it needs to walk the directory tree and any symlinks to temporary/intermediate locations should have the underlying files moved into the output.

[mr-c]

FYI: this still occurs as of cwltool 3.1.20210922203925