Skip to content

Commit 5d34eb8

Browse files
authored
Merge branch 'main' into ci_apptainer
2 parents 25824f0 + b0838ea commit 5d34eb8

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

55 files changed

+2126
-515
lines changed

.flake8

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
[flake8]
2+
ignore = E203,W503
3+
max-line-length = 100
4+
select = B,C,E,F,W,T4
5+
exclude = cwltool/schemas
6+
extend-ignore = E501,B905
7+
# when Python 3.10 is the minimum version, re-enable check B905 for zip + strict
8+
extend-select = B9

.git-blame-ignore-revs

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# black + isort
2+
69e709fde4a3afada10d441cb9e0a85f8a95a3d4

.github/workflows/ci-tests.yml

Lines changed: 9 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ jobs:
5757
- name: Upgrade setuptools and install tox
5858
run: |
5959
pip install -U pip setuptools wheel
60-
pip install tox tox-gh-actions
60+
pip install "tox<4" "tox-gh-actions<3"
6161
6262
- name: MyPy cache
6363
if: ${{ matrix.step == 'mypy' }}
@@ -86,8 +86,8 @@ jobs:
8686
step: [lint-readme, pydocstyle]
8787

8888
env:
89-
py-semver: 3.11
90-
TOXENV: ${{ format('py310-{0}', matrix.step) }}
89+
py-semver: "3.11"
90+
TOXENV: ${{ format('py311-{0}', matrix.step) }}
9191

9292
steps:
9393
- uses: actions/checkout@v3
@@ -98,17 +98,12 @@ jobs:
9898
uses: actions/setup-python@v4
9999
with:
100100
python-version: ${{ env.py-semver }}
101-
102-
- name: Cache for pip
103-
uses: actions/cache@v3
104-
with:
105-
path: ~/.cache/pip
106-
key: ${{ runner.os }}-pip-${{ matrix.step }}-${{ hashFiles('requirements.txt') }}
101+
cache: pip
107102

108103
- name: Upgrade setuptools and install tox
109104
run: |
110105
pip install -U pip setuptools wheel
111-
pip install tox tox-gh-actions
106+
pip install "tox<4" "tox-gh-actions<3"
112107
113108
- if: ${{ matrix.step == 'pydocstyle' && github.event_name == 'pull_request'}}
114109
name: Create local branch for diff-quality for PRs
@@ -119,7 +114,6 @@ jobs:
119114

120115
release_test:
121116
name: cwl-utils release test
122-
123117
runs-on: ubuntu-22.04
124118

125119
steps:
@@ -134,12 +128,10 @@ jobs:
134128
uses: actions/setup-python@v4
135129
with:
136130
python-version: 3.11
137-
138-
- name: Cache for pip
139-
uses: actions/cache@v3
140-
with:
141-
path: ~/.cache/pip
142-
key: ${{ runner.os }}-pip-release-${{ hashFiles('requirements.txt', 'test-requirements.txt') }}
131+
cache: pip
132+
cache-dependency-path: |
133+
requirements.txt
134+
test-requirements.txt
143135
144136
- name: Install packages
145137
run: |

.readthedocs.yml

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,9 +12,14 @@ sphinx:
1212
# Optionally build your docs in additional formats such as PDF and ePub
1313
formats: all
1414

15-
# Optionally set the version of Python and requirements required to build your docs
15+
build:
16+
os: ubuntu-22.04
17+
tools:
18+
python: "3.11"
19+
apt_packages:
20+
- graphviz
21+
1622
python:
17-
version: 3.7
1823
install:
1924
- requirements: docs/requirements.txt
2025
- method: pip

CONTRIBUTING.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,9 @@ Style guide:
33
- Python 3.6+ compatible code
44
- PEP-484 type hints
55

6+
It is suggested that you run `git config blame.ignoreRevsFile .git-blame-ignore-revs`
7+
to filter out mass-formatting commits from `git blame`.
8+
69
In order to contribute to the development of ``cwl-utils``, you need to install cwl-utils from source (preferably in a virtual environment):
710
Here's a rough guide (improvements are welcome!)
811
- Install virtualenv via pip: ``pip install virtualenv``

MANIFEST.in

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,9 @@ include testdata/*.yaml
1313
include testdata/*.input
1414
include testdata/*.ttl
1515
include testdata/*.owl
16+
include testdata/checker_wf/*.cwl
1617
include cwl_utils/py.typed
18+
include docs/conf.py docs/Makefile docs/_static/favicon.ico docs/requirements.txt
19+
include docs/*.rst
1720
global-exclude *~
1821
global-exclude *.pyc

Makefile

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ EXTRAS=
2828
# `[[` conditional expressions.
2929
PYSOURCES=$(filter-out $(MODULE)/parser/cwl_v%,$(shell find $(MODULE) -name "*.py")) \
3030
$(wildcard tests/*.py) create_cwl_from_objects.py load_cwl_by_path.py \
31-
setup.py ${MODULE}/parser/cwl_v1_?_utils.py
31+
setup.py ${MODULE}/parser/cwl_v1_?_utils.py docs/conf.py
3232
DEVPKGS=diff_cover black pylint pep257 pydocstyle flake8 tox tox-pyenv \
3333
isort wheel autoflake flake8-bugbear pyupgrade bandit \
3434
-rtest-requirements.txt -rmypy-requirements.txt
@@ -109,8 +109,8 @@ codespell:
109109
codespell -w $(shell git ls-files | grep -v mypy-stubs)
110110

111111
## format : check/fix all code indentation and formatting (runs black)
112-
format: $(PYSOURCES)
113-
black $^
112+
format: $(PYSOURCES) FORCE
113+
black $(PYSOURCES)
114114

115115
format-check: $(PYSOURCES)
116116
black --diff --check $^

cwl_utils/__meta__.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
# SPDX-License-Identifier: Apache-2.0
22
"""Global version number for the cwl_utils package."""
3-
__version__ = "0.21"
3+
__version__ = "0.22"

cwl_utils/cite_extract.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010

1111

1212
def arg_parser() -> argparse.ArgumentParser:
13-
"""Argument parser."""
13+
"""Construct the argument parser."""
1414
parser = argparse.ArgumentParser(
1515
description="Print information about software used in a CWL document (Workflow or CommandLineTool). "
1616
"For CWL Workflows, all steps will also be searched (recursively)."

cwl_utils/cwl_v1_0_expression_refactor.py

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -350,7 +350,7 @@ def generate_etool_from_expr(
350350
elif self_type.type:
351351
new_type = clean_type_ids(self_type.type)
352352
else:
353-
raise WorkflowException(f"Don't know how to make type from '{self_type}'.")
353+
raise WorkflowException(f"Don't know how to make type from {self_type!r}.")
354354
inputs.append(
355355
cwl.InputParameter(
356356
id="self",
@@ -652,7 +652,8 @@ def process_workflow_reqs_and_hints(
652652
"""
653653
# TODO: consolidate the generated etools/cltools into a single "_expression_workflow_reqs" step
654654
# TODO: support resourceReq.* references to Workflow.inputs?
655-
# ^ By refactoring replace_expr_etool to allow multiple inputs, and connecting all workflow inputs to the generated step
655+
# ^ By refactoring replace_expr_etool to allow multiple inputs,
656+
# and connecting all workflow inputs to the generated step
656657
modified = False
657658
inputs = empty_inputs(workflow)
658659
generated_res_reqs: List[Tuple[str, Union[int, str]]] = []
@@ -837,7 +838,8 @@ def process_workflow_reqs_and_hints(
837838
raise_type=WorkflowException,
838839
).makeError(
839840
f"Entry {index},"
840-
+ "Invalid CWL, if 'entry' is a string, then entryName must be specified."
841+
+ "Invalid CWL, if 'entry' "
842+
"is a string, then entryName must be specified."
841843
)
842844
expr = (
843845
'${return {"class": "File", "basename": "'
@@ -979,7 +981,8 @@ def process_level_reqs(
979981
# - Replace the CWL Expression inplace with a CWL parameter reference
980982
# - Don't create a new Requirement, nor delete the existing Requirement
981983
# - the Process is passed to replace_expr_with_etool for later searching for JS expressionLibs
982-
# - in addition to adding the input to the step for the ETool/CTool result, add it to the Process.inputs as well
984+
# - in addition to adding the input to the step for the ETool/CTool result,
985+
# add it to the Process.inputs as well
983986
if not process.requirements:
984987
return False
985988
modified = False
@@ -1122,7 +1125,7 @@ def process_level_reqs(
11221125
entryname = (
11231126
entry.entryname
11241127
if entryname_expr
1125-
else f'"{entry.entryname}"'
1128+
else f'"{entry.entryname}"' # noqa: B907
11261129
)
11271130
new_expression = (
11281131
"${var result; var entryname = "
@@ -1914,15 +1917,15 @@ def workflow_step_to_InputParameters(
19141917
for p in param:
19151918
if not p.type:
19161919
raise WorkflowException(
1917-
f"Don't know how to get type id for '{p}'."
1920+
f"Don't know how to get type id for {p!r}."
19181921
)
19191922
p.id = inp_id
19201923
p.type = clean_type_ids(p.type)
19211924
params.append(p)
19221925
else:
19231926
if not param.type:
19241927
raise WorkflowException(
1925-
f"Don't know how to get type id for '{param}'."
1928+
f"Don't know how to get type id for {param!r}."
19261929
)
19271930
param.id = inp_id
19281931
param.type = clean_type_ids(param.type)

cwl_utils/cwl_v1_1_expression_refactor.py

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -183,7 +183,7 @@ def etool_to_cltool(
183183
contents += "\n" + "\n".join(expressionLib)
184184
contents += (
185185
"""
186-
var ret = function(){"""
186+
var ret = function(){"""
187187
+ escape_expression_field(etool.expression.strip()[2:-1])
188188
+ """}();
189189
process.stdout.write(JSON.stringify(ret));"""
@@ -389,7 +389,7 @@ def generate_etool_from_expr(
389389
expression += "\n var self=inputs.self;"
390390
expression += (
391391
"""
392-
return {"result": function(){"""
392+
return {"result": function(){"""
393393
+ expr[2:-2]
394394
+ """}()};
395395
}"""
@@ -652,7 +652,8 @@ def process_workflow_reqs_and_hints(
652652
"""
653653
# TODO: consolidate the generated etools/cltools into a single "_expression_workflow_reqs" step
654654
# TODO: support resourceReq.* references to Workflow.inputs?
655-
# ^ By refactoring replace_expr_etool to allow multiple inputs, and connecting all workflow inputs to the generated step
655+
# ^ By refactoring replace_expr_etool to allow multiple inputs,
656+
# and connecting all workflow inputs to the generated step
656657
modified = False
657658
inputs = empty_inputs(workflow)
658659
generated_res_reqs: List[Tuple[str, Union[int, str]]] = []
@@ -837,7 +838,8 @@ def process_workflow_reqs_and_hints(
837838
raise_type=WorkflowException,
838839
).makeError(
839840
f"Entry {index},"
840-
+ "Invalid CWL, if 'entry' is a string, then entryName must be specified."
841+
+ "Invalid CWL, if 'entry' "
842+
"is a string, then entryName must be specified."
841843
)
842844
expr = (
843845
'${return {"class": "File", "basename": "'
@@ -979,7 +981,8 @@ def process_level_reqs(
979981
# - Replace the CWL Expression inplace with a CWL parameter reference
980982
# - Don't create a new Requirement, nor delete the existing Requirement
981983
# - the Process is passed to replace_expr_with_etool for later searching for JS expressionLibs
982-
# - in addition to adding the input to the step for the ETool/CTool result, add it to the Process.inputs as well
984+
# - in addition to adding the input to the step for the ETool/CTool result,
985+
# add it to the Process.inputs as well
983986
if not process.requirements:
984987
return False
985988
modified = False
@@ -1122,7 +1125,7 @@ def process_level_reqs(
11221125
entryname = (
11231126
entry.entryname
11241127
if entryname_expr
1125-
else f'"{entry.entryname}"'
1128+
else f'"{entry.entryname}"' # noqa: B907
11261129
)
11271130
new_expression = (
11281131
"${var result; var entryname = "
@@ -1716,7 +1719,7 @@ def generate_etool_from_expr2(
17161719
expression += f"\n var self=inputs.{self_name};"
17171720
expression += (
17181721
"""
1719-
return {"result": function(){"""
1722+
return {"result": function(){"""
17201723
+ expr[2:-2]
17211724
+ """}()};
17221725
}"""

cwl_utils/cwl_v1_2_expression_refactor.py

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -183,7 +183,7 @@ def etool_to_cltool(
183183
contents += "\n" + "\n".join(expressionLib)
184184
contents += (
185185
"""
186-
var ret = function(){"""
186+
var ret = function(){"""
187187
+ escape_expression_field(etool.expression.strip()[2:-1])
188188
+ """}();
189189
process.stdout.write(JSON.stringify(ret));"""
@@ -389,7 +389,7 @@ def generate_etool_from_expr(
389389
expression += "\n var self=inputs.self;"
390390
expression += (
391391
"""
392-
return {"result": function(){"""
392+
return {"result": function(){"""
393393
+ expr[2:-2]
394394
+ """}()};
395395
}"""
@@ -747,7 +747,8 @@ def process_workflow_reqs_and_hints(
747747
"""
748748
# TODO: consolidate the generated etools/cltools into a single "_expression_workflow_reqs" step
749749
# TODO: support resourceReq.* references to Workflow.inputs?
750-
# ^ By refactoring replace_expr_etool to allow multiple inputs, and connecting all workflow inputs to the generated step
750+
# ^ By refactoring replace_expr_etool to allow multiple inputs,
751+
# and connecting all workflow inputs to the generated step
751752
modified = False
752753
inputs = empty_inputs(workflow)
753754
generated_res_reqs: List[Tuple[str, Union[int, str]]] = []
@@ -932,7 +933,8 @@ def process_workflow_reqs_and_hints(
932933
raise_type=WorkflowException,
933934
).makeError(
934935
f"Entry {index},"
935-
+ "Invalid CWL, if 'entry' is a string, then entryName must be specified."
936+
+ "Invalid CWL, if 'entry' "
937+
"is a string, then entryName must be specified."
936938
)
937939
expr = (
938940
'${return {"class": "File", "basename": "'
@@ -1074,7 +1076,8 @@ def process_level_reqs(
10741076
# - Replace the CWL Expression inplace with a CWL parameter reference
10751077
# - Don't create a new Requirement, nor delete the existing Requirement
10761078
# - the Process is passed to replace_expr_with_etool for later searching for JS expressionLibs
1077-
# - in addition to adding the input to the step for the ETool/CTool result, add it to the Process.inputs as well
1079+
# - in addition to adding the input to the step for the ETool/CTool result,
1080+
# add it to the Process.inputs as well
10781081
if not process.requirements:
10791082
return False
10801083
modified = False
@@ -1217,7 +1220,7 @@ def process_level_reqs(
12171220
entryname = (
12181221
entry.entryname
12191222
if entryname_expr
1220-
else f'"{entry.entryname}"'
1223+
else f'"{entry.entryname}"' # noqa: B907
12211224
)
12221225
new_expression = (
12231226
"${var result; var entryname = "
@@ -1811,7 +1814,7 @@ def generate_etool_from_expr2(
18111814
expression += f"\n var self=inputs.{self_name};"
18121815
expression += (
18131816
"""
1814-
return {"result": function(){"""
1817+
return {"result": function(){"""
18151818
+ expr[2:-2]
18161819
+ """}()};
18171820
}"""

cwl_utils/errors.py

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,23 +3,23 @@
33

44

55
class ArrayMissingItems(BaseException):
6-
"""From https://github.com/rabix/sbpack/blob/b8404a0859ffcbe1edae6d8f934e51847b003320/sbpack/lib.py"""
6+
"""From https://github.com/rabix/sbpack/blob/b8404a0859ffcbe1edae6d8f934e51847b003320/sbpack/lib.py ."""
77

88

99
class JavascriptException(Exception):
1010
pass
1111

1212

1313
class MissingKeyField(BaseException):
14-
"""From https://github.com/rabix/sbpack/blob/b8404a0859ffcbe1edae6d8f934e51847b003320/sbpack/lib.py"""
14+
"""From https://github.com/rabix/sbpack/blob/b8404a0859ffcbe1edae6d8f934e51847b003320/sbpack/lib.py ."""
1515

1616

1717
class MissingTypeName(BaseException):
18-
"""From https://github.com/rabix/sbpack/blob/b8404a0859ffcbe1edae6d8f934e51847b003320/sbpack/lib.py"""
18+
"""From https://github.com/rabix/sbpack/blob/b8404a0859ffcbe1edae6d8f934e51847b003320/sbpack/lib.py ."""
1919

2020

2121
class RecordMissingFields(BaseException):
22-
"""From https://github.com/rabix/sbpack/blob/b8404a0859ffcbe1edae6d8f934e51847b003320/sbpack/lib.py"""
22+
"""From https://github.com/rabix/sbpack/blob/b8404a0859ffcbe1edae6d8f934e51847b003320/sbpack/lib.py ."""
2323

2424

2525
class SubstitutionError(Exception):

0 commit comments

Comments
 (0)