Skip to content

Latest commit

 

History

History
140 lines (107 loc) · 5.99 KB

CONTRIBUTING.md

File metadata and controls

140 lines (107 loc) · 5.99 KB

Contribute to StreamFlow

As a first step, clone StreamFlow from GitHub

git clone git@github.com:alpha-unito/streamflow.git

Then, install all the required packages using the pip install command

cd streamflow
pip install -r requirements.txt

Finally, install StreamFlow with the following command

pip install .

Continuous Integration

StreamFlow relies on GitHub Actions for Continuous Integration (CI) and Continuous Distribution (CD). Only maintainers take care of the CD pipeline. In order to publish a new version of the software on PyPI and Docker Hub distributions, a maintainer only has to augment the version number in the version.py file.

Instead, everyone in the community can contribute to the StreamFlow codebase by opening a Pull Request (PR). Running the entire suite of StreamFlow tests is part of the CI pipeline. However, it is also possible (and advisable) to run tests locally before opening a PR, as explained below.

CWL conformance

StreamFlow complies with all stable versions of the Common Workflow Language (CWL) open standard (v1.0, v1.1, and v1.2). Plus, it implements the cwltool:Loop extension (see here). You can check if your PR does not compromise CWL compliance by running the CWL conformance tests suite, using the following command

./cwl-conformance-test.sh

StreamFlow tests

Some non regression tests are supplied using pytest. Install the required packages using the following command

pip install -r test-requirements.txt

Then, you can execute all the available tests with the following command

make test

Otherwise, you can execute only a specific test file. For example, to verify compliance with the cwltool:Loop CWL extension, you can use the following command

pytest tests/test_cwl_loop.py

Testing connectors

StreamFlow comes with many different connectors OOTB, and some of the tests in the suite are used to verify their behaviour. Currently, the tested connectors are: local, docker, docker-compose, ssh, slurm, kubernetes, and singularity. The plan is to add non regression tests for all connectors. Executing all these tests requires to install and configure several software packages:

  • Docker, which is required to test the docker, docker-compose, slurm, and ssh connectors and the CWL DockerRequirement implementation with Docker containers;
  • Singularity, which is required to test the singularity connector and the CWL DockerRequirement conversion to Singularity containers;
  • A Kubernetes distribution, e.g., minikube, which is used to test the kubernetes connector and the CWL DockerRequirement conversion to Kubernetes pods.

For development purposes, it is possible to execute local tests only on a specific subset of connectors using the --deploys option as follows

pytest --deploys local,docker,singularity tests/test_remotepath.py

However, all connectors are always tested in the CI pipeline.

Code style

StreamFlow supports Python 3.9 to 3.13, so use compatible code features.

The StreamFlow code style complies with PEP 8. Some automatic style checkers can be installed with

pip install -r lint-requirements.txt

You can verify that your PR respects the code style with the following command

make format-check flake8 codespell-check

You can also apply the suggested changes in bulk with the following command

make format codespell

Use it carefully as it modifies the target files in place.

Finally, the StreamFlow documentation relies on the Sphinx framework, which follows the reStructuredText format. Follow this style to add documentation for new functions.

This is a reST style.

:param param1: this is a first param
:param param2: this is a second param
:returns: this is a description of what is returned
:raises keyError: raises an exception

Contribute to the documentation

Generate the documentation

The documentation is written in the reStructuredText markup language, and the Sphinx framework is used to generate the HTML pages published on the StreamFlow website. Currently, there is no CD for documentation: maintainers manually transfer files on the StreamFlow website at every release.

To check your local version of the documentation, first, install all the required packages using the following command

cd docs
pip install -r requirements.txt

Then, generate the documentation in HTML format with the following command

make html

If you want, you can also generate documentation in a different format from the ones supported by Sphinx. However, note that StreamFlow officially maintains only the HTML format. For more information on the available documentation formats, use the following command

make help

Documentation changes

Whenever a commit changes the documentation, it is necessary to update the dedicated test on the CI pipeline. The checksum of the documentation in HTML format is tested in the CI pipeline to avoid unwanted changes when Sphinx or other dependencies are updated.

The command below prints the new checksum on the standard output

cd docs
make checksum

This new checksum must coincide with the one in the .github/workflows/ci-tests.yaml file, i.e., the value of the CHECKSUM environment variable in the documentation job

name: "CI Tests"
jobs:
  documentation:
    name: "Build Sphinx documentation"
    steps:
      env:
        CHECKSUM: "NEW_CHECKSUM"