--skip-schemas isn't stopping cwltool from trying to connect to the schema server

[douglowe]

Expected Behavior

When I use cwltool --skip-schemas on a system with no internet access, the workflow should still be runnable.

Actual Behavior

When I use cwltool --skip-schemas the process hangs at the Resolved [workflow] to [path to workflow] step. I have to hard-kill the process to stop it. When I delete all the schema information in my script though ($schemas and the format instructions which depend on these), the script works as expected.

The code below (provided by @mr-c) replicates this problem for systems which have internet access.

Workflow Code

wget https://github.com/bioexcel/biobb_adapters/raw/master/biobb_adapters/cwl/biobb_md/mdrun.cwl
HTTPS_PROXY=http://127.0.0.5 cwltool --debug --skip-schemas --validate mdrun.cwl

Full Traceback

(cwl) 192:test mbessdl2$ HTTPS_PROXY=http://127.0.0.5 cwltool --debug --skip-schemas --validate mdrun.cwl
INFO /Users/mbessdl2/anaconda3/envs/cwl/bin/cwltool 3.0.20200807132242
INFO Resolved 'mdrun.cwl' to 'file:///Users/mbessdl2/work/manchester/BioExcel/workflows_to_adapt/biobb_hpc_cwl_md_list/test/mdrun.cwl'
^CTraceback (most recent call last):
  File "/Users/mbessdl2/anaconda3/envs/cwl/bin/cwltool", line 11, in <module>
    sys.exit(run())
  File "/Users/mbessdl2/anaconda3/envs/cwl/lib/python3.8/site-packages/cwltool/main.py", line 1267, in run
    sys.exit(main(*args, **kwargs))
  File "/Users/mbessdl2/anaconda3/envs/cwl/lib/python3.8/site-packages/cwltool/main.py", line 939, in main
    loadingContext, uri = resolve_and_validate_document(
  File "/Users/mbessdl2/anaconda3/envs/cwl/lib/python3.8/site-packages/cwltool/load_tool.py", line 378, in resolve_and_validate_document
    document_loader.resolve_all(workflowobj, fileuri)
  File "/Users/mbessdl2/anaconda3/envs/cwl/lib/python3.8/site-packages/schema_salad/ref_resolver.py", line 957, in resolve_all
    loader.validate_links(
  File "/Users/mbessdl2/anaconda3/envs/cwl/lib/python3.8/site-packages/schema_salad/ref_resolver.py", line 1107, in validate_links
    document[d] = self.validate_link(
  File "/Users/mbessdl2/anaconda3/envs/cwl/lib/python3.8/site-packages/schema_salad/ref_resolver.py", line 1063, in validate_link
    link[n] = self.validate_link(field, i, docid, all_doc_ids)
  File "/Users/mbessdl2/anaconda3/envs/cwl/lib/python3.8/site-packages/schema_salad/ref_resolver.py", line 1053, in validate_link
    elif not self.check_exists(link):
  File "/Users/mbessdl2/anaconda3/envs/cwl/lib/python3.8/site-packages/schema_salad/fetcher.py", line 93, in check_exists
    resp = self.session.head(url)
  File "/Users/mbessdl2/anaconda3/envs/cwl/lib/python3.8/site-packages/requests/sessions.py", line 565, in head
    return self.request('HEAD', url, **kwargs)
  File "/Users/mbessdl2/anaconda3/envs/cwl/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/mbessdl2/anaconda3/envs/cwl/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/Users/mbessdl2/anaconda3/envs/cwl/lib/python3.8/site-packages/cachecontrol/adapter.py", line 47, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "/Users/mbessdl2/anaconda3/envs/cwl/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/Users/mbessdl2/anaconda3/envs/cwl/lib/python3.8/site-packages/urllib3/connectionpool.py", line 667, in urlopen
    self._prepare_proxy(conn)
  File "/Users/mbessdl2/anaconda3/envs/cwl/lib/python3.8/site-packages/urllib3/connectionpool.py", line 932, in _prepare_proxy
    conn.connect()
  File "/Users/mbessdl2/anaconda3/envs/cwl/lib/python3.8/site-packages/urllib3/connection.py", line 309, in connect
    conn = self._new_conn()
  File "/Users/mbessdl2/anaconda3/envs/cwl/lib/python3.8/site-packages/urllib3/connection.py", line 159, in _new_conn
    conn = connection.create_connection(
  File "/Users/mbessdl2/anaconda3/envs/cwl/lib/python3.8/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
KeyboardInterrupt

Your Environment

• cwltool version: 3.0.20200807132242
  (however @mr-c has checked this on more recent versions of cwltool, and it is still an issue for these too)

[mr-c]

@douglowe I forgot to wget the file and then validate it; the issue is not present for me with the latest cwltool version (and the conda package appears to be fixed now

$ HTTPS_PROXY=http://127.0.0.5 cwltool --debug --skip-schemas --validate mdrun.cwl 
INFO /home/michael/cwltool/env3.9/bin/cwltool 3.1.20210317115921
INFO Resolved 'mdrun.cwl' to 'file:///home/michael/cwltool/mdrun.cwl'
Warning: Field `$schemas` contains undefined reference to `https://raw.githubusercontent.com/edamontology/edamontology/master/EDAM_dev.owl`
mdrun.cwl:126:3: object id `mdrun.cwl#output_cpt_path` previously defined
mdrun.cwl:141:3: object id `mdrun.cwl#output_dhdl_path` previously defined
mdrun.cwl:65:3: object id `mdrun.cwl#output_edr_path` previously defined
mdrun.cwl:49:3: object id `mdrun.cwl#output_gro_path` previously defined
mdrun.cwl:81:3: object id `mdrun.cwl#output_log_path` previously defined
mdrun.cwl:33:3: object id `mdrun.cwl#output_trr_path` previously defined
mdrun.cwl:111:3: object id `mdrun.cwl#output_xtc_path` previously defined
mdrun.cwl is valid CWL.

[douglowe]

Ahh, good.

Okay, I've managed to update to the latest version (3.0.20210124104916) with the updated schema-salad package, and it does work for me too now. There is still a long delay though, presumably as cwltool tries to connect to https://raw.githubusercontent.com/edamontology/edamontology/master/EDAM_dev.owl. Should cwltool still try to connect to this if --skip-schemas has been specified?

[mr-c]

Should cwltool still try to connect to this if --skip-schemas has been specified?

Yes, that would be a good thing to fix