feat(launcher): prepare python library (#18)

Feature:

* add a (modern) configuration file parser to search the configuration file in the user configuration directory, or in the current working directory.
* change the launcher scripts to use the configuration object.

Bug fixes:

* bad empty string comparison in `RemoteServerCleaner._final_zip_downloaded`.
* add the missing `json_dir` in the configuration.

Build:

* upgrade the `setup.py` to create a Python library.
* correct the `MANIFEST.in` file used for source distribution.
* the minimum supported version of Python is 3.8 (older versions are not guaranteed to work).
* correct Project metadata (version, description, etc.).
* update/regenerate `requirements.txt` files.

Test:

* Remove the end-to-end tests.

Documentations:

* improve the configuration documentation in unit tests.
* update the project dependencies.
* update and correct the documentation.
* remove `standards.md` (not used).
* correct the documentation about pyinstaller.

MANIFEST.in

@@ -1,7 +1,23 @@
+# This configuration file lists the elements to include in the source
+# distribution of the Python package.
+# See: https://packaging.python.org/en/latest/guides/using-manifest-in/
+
+graft antareslauncher
+graft doc
+graft remote_scripts_templates
+graft tests
+
+include *.ini
+include *.md
+include *.py
+include *.txt
+include Makefile
+
 # The `data` directory is intended to store your configuration for integration testing with a SLURM server.
 # It should contain the application configuration file `configuration.yaml` and the SSH configuration file
 # `ssh_config.json`. The files in this directory should never be stored in the source distribution (except `README.md`).
-include data/
 include data/.gitignore
 include data/README.md
 recursive-exclude data *
+
+global-exclude *.py[cod] .DS_Store

README.md

@@ -1,54 +1,108 @@
 # Antares Launcher
 
-This program is meant to allow the user to send a list of Antares simulations
-on a remote Linux machine that can run them using *SLURM Workload Manager*.
+This program is intended to allow the user to send a list of Antares simulations to
+a remote Linux machine that can run them using *SLURM Workload Manager*.
 
-At present this program:
+Currently, this program:
 
-- is configured to work with Antares studies of version 6.1.3 and 7.0.0 (the configuration can be changed in YAML file)
-- needs a remote unix server that uses *SLURM Workload Manager*
+- Is configured to work with Antares studies from version 7.0 through 8.5
+  (the configuration can be changed in a YAML file).
+- needs a remote UNIX server that uses *SLURM Workload Manager*.
 
-A schema of the main workflow is as follow
+The main workflow diagram is as follows:
 
-![Antares Study Launcher](./doc/source/schema/antares_flow_chart_AS-FINAL-withbranch-wait.png)
+![Antares Study Launcher](https://raw.githubusercontent.com/AntaresSimulatorTeam/antares-launcher/main/doc/source/schema/antares_flow_chart_AS-FINAL-withbranch-wait.png)
 
-### Requirements
-see [Requirements file](./requirements.txt)
-minimum version : python 3.6
+## Requirements
 
-paramiko, pytest, pytest-cov, pytest-ordering, tinydb, black, pyinstaller, setuptools==44.1.1, tqdm
-pytest for the tests
-pyinstaller to 'compile' the project into binary file
+See [`setup.py`](https://github.com/AntaresSimulatorTeam/antares-launcher/blob/main/setup.py)
 
-#### For documentation
-sphinx
+Minimum version : python 3.8
 
-##### To convert the doc from markdown to sphinx (rst)
-m2r
+### Main Python libraries
+
+The following libraries are required to run the application in a production (or staging) environment:
+
+- paramiko
+- PyYAML
+- tinydb
+- tqdm
+
+To install this library on production, you can run:
+
+```shell
+pip install Antares-Launcher
+```
+
+### Development and Unit Testing
+
+To start developing, you can clone the repository from GitHub and create a Python virtualenv:
+
+```shell
+cd ~/workspace/
+git clone https://github.com/AntaresSimulatorTeam/antares-launcher.git
+cd ~/workspace/antares-launcher/
+python3 -m venv venv
+source venv/bin/activate
+```
+
+To run the unit tests, you need to install:
+
+- pytest
+- pytest-cov
+- pytest-xdist
+
+To install this library in development mode for testing, you can run:
+
+```shell
+pip install -e .[test]
+```
+
+Additional dependencies could also be used for development, for instance:
+
+- black
+- check-manifest
+- isort
+- mypy
+
+### Documentation
+
+In this project, we use Sphinx to generate the documentation.
+
+Extra requirements are:
+
+- m2r
+- recommonmark 
+- sphinx
+- sphinx_rtd_theme 
 
 ## Installation
 
 ### Generation of the binary executable
+
 In order to generate the binary file, execute the following command:
 
 ```
 pyinstaller --additional-hooks-dir=antareslauncher/hooks/ -F antareslauncher/main_launcher.py -n Antares_Launcher
 ```
 
-In order to generate the binary file of the light version of the launcher (reduced set of options), execute the following command:
+In order to generate the binary file of the light version of the launcher (reduced set of options), execute the
+following command:
 
 ```
 pyinstaller --additional-hooks-dir=antareslauncher/hooks/ -F antareslauncher/main_launcher_light.py -n Antares_Launcher_Light
 ```
 
-The generated file will be inside the dist directory. Note that pyinstaller does not enable the cross-compilation: e binary file generated on windows can only be expected with the windows OS
+The generated file will be inside the dist directory. Note that pyinstaller does not enable the cross-compilation: e
+binary file generated on windows can only be expected with the windows OS
 
 ## Use Antares_Launcher
 
 ### Run Antares_Launcher
+
 **Antares Launcher** can be used by running the executable file
 
-By default the program will
+By default, the program will:
 
 - look for a configuration file necessary for the connection
 named *ssh_config.json*.
@@ -60,26 +114,27 @@ If no value is given, it will look for it in default location with this order:
 A default *ssh_config.json* file can be found in this
 repository in the `./data` directory of the project
 
-
 - look for an rsa-private ssh-key to access to the remote server.
 The path of the key is specified in the `ssh_config.json` file
 
 - look for a directory containing
-the Antares studies to be run on the remote machine
-named *STUDIES-IN*.
+  the Antares studies to be run on the remote machine
+  named *STUDIES-IN*.
 
 - put the results in the directory named
-*FINISHED*
+  *FINISHED*
 
 - create a directory *LOGS* that contains the logs of the programs
-and several directories containing the three log files specific of each simulation.  
-Currently **antares_launcher** uses a specific configuration attached to the specific setting of
-`data/launchAntares-${SCRIPT-VERSION}.sh`
+  and several directories containing the three log files specific of each simulation.  
+  Currently **antares_launcher** uses a specific configuration attached to the specific setting of
+  `data/launchAntares-${SCRIPT-VERSION}.sh`
 
 #### Get the *how-to*
+
 ```
 Antares_Launcher --help
 ```
+
 will show how to use the program.
 
 ### SLURM script on the remote machine
@@ -88,34 +143,43 @@ In order to submit new jobs to the *SLURM* queue manager,
 **Antares_Launcher** launches a bash-SLURM script the name of the script is set in `data/configuration.yaml`.
 If Antares_Launcher fails to find this script
 an exception will be raised and the execution will stop.
- 
+
 The specification of the script can be found in the class
 `SlurmScriptFeatures` in the module `antareslauncher.slurm_script_features.py`.
 See [Deploy Antares Launcher](#deploy-antares-launcher) for specific values.
 
 ## Useful commands
+
 Since the addition of the Makefile to the project, one can now easily set a virtual environment, install requirements,
 generate binary file, run tests, generate the doc and deploy it...
 
 At the root of the directory, all the available commands can be seen with typing: make
 
-![Antares Study Launcher](./doc/source/schema/make_example.png)
+![Antares Study Launcher](https://raw.githubusercontent.com/AntaresSimulatorTeam/antares-launcher/main/doc/source/schema/make_example.png)
 
 If for example, you would like to run the test, a simple ``make test`` will do the trick
 
-![Antares Study Launcher](./doc/source/schema/make_test_example.png)
+![Antares Study Launcher](https://raw.githubusercontent.com/AntaresSimulatorTeam/antares-launcher/main/doc/source/schema/make_test_example.png)
 
-## More useful commands
-`pytest --cov=../antareslauncher`
+## Useful commands
 
-`pytest --cov=../antareslauncher --cov-report term-missing`
+Run unit tests:
 
-`ssh [email protected]`
+```shell
+pytest -v tests/
+```
 
-`pytest --cov=../antareslauncher --cov-report term-missing -p no:warnings`
+Run unit tests with code coverage:
+
+```shell
+pytest --cov=antareslauncher --cov-report=term-missing --cov-report=html --cov-branch tests/
+open htmlcov/index.html
+```
 
 # Deploy Antares Launcher
+
 ## Installation on the remote server
+
 In order to be able to accept jobs from Antares_Launcher, the remote machine needs to be ready:
 the binaries and script expected by **Antares_Launcher** need to be installed and
 the required ssh-public-keys need to be added to the `authorizedkeys` file
@@ -124,22 +188,24 @@ of the account of the remote server.
 ### Things to do
 
 - `launchAntares-${SCRIPT-VERSION}.sh` should be copied to the remove server
-and ist path should be set in `data/configuration.yaml` 
+  and ist path should be set in `data/configuration.yaml`
 
-- Install the Antares solver binary `antares-x.x-solver` on the remote server. 
-set its installation path in `launchAntares-${SCRIPT-VERSION}.sh`
+- Install the Antares solver binary `antares-x.x-solver` on the remote server.
+  set its installation path in `launchAntares-${SCRIPT-VERSION}.sh`
 
 - The R Xpansion script, `data/XpansionArgsRun.R`,
-has to be copied to the remote server and
- its path should be set in `launchAntares-${SCRIPT-VERSION}.sh`.
- 
+  has to be copied to the remote server and
+  its path should be set in `launchAntares-${SCRIPT-VERSION}.sh`.
+
 #### Important notice
+
 The users currently copy the executable every time they need to use it.
-This is not practical, an alternative should be developed. 
+This is not practical, an alternative should be developed.
 
 ## Installation of R packages on the remote server
-In order to correctly install or update packages to be used on the remote server 
+
+In order to correctly install or update packages to be used on the remote server
 the *R*repositories and installation-destination need to be set.
 
 The `launchAntares-${SCRIPT-VERSION}.sh` set the variable where the *R*libraries are installed runtime,
- no need to create a `.Renviron` file.
+no need to create a `.Renviron` file.

antareslauncher/__init__.py

@@ -1,8 +1,49 @@
-#: Project's name (str)
-PROJECT_NAME = "Antares_Launcher"
+"""
+Antares Launcher
 
-#: Project's description (str)
-DESCRIPTION = "Antares_Launcher to run Antares on a remote linux machine"
+This program is meant to allow the user to send a list of Antares simulations
+on a remote Linux machine that can run them using *SLURM Workload Manager*.
 
-#: Project's version (:py:class:`Version <rspace.application.Version.Version>`)
-VERSION = "1.1.3"
+This module contains the project metadata.
+"""
+
+# Standard project metadata
+
+__version__ = "1.1.4"
+__author__ = "RTE, Antares Web Team"
+__date__ = "2021-05-25"
+# noinspection SpellCheckingInspection
+__credits__ = "(c) Réseau de Transport de l’Électricité (RTE)"
+
+# Extra project metadata
+__project_name__ = "Antares_Launcher"
+
+
+def _check_metadata():
+    # noinspection SpellCheckingInspection
+    """
+    Check the project metadata.
+
+    To update the project metadata, you can run the following command:
+    ```shell
+    python setup.py egg_info
+    ```
+
+    To get the list of tags and release dates, you can also run:
+    ```shell
+    git for-each-ref --sort=-creatordate --format '%(refname:strip=2) (%(creatordate:short))' refs/tags
+    ```
+    """
+    from pkg_resources import get_distribution
+
+    dist = get_distribution(__project_name__)
+    # Expected distribution name should be "antares-launcher":
+    dist_name = __project_name__.lower().replace("_", "-")
+    assert dist.key == dist_name, dist.key
+    assert dist.version == __version__, dist.version
+    print("Project metadata OK.")
+
+
+if __name__ == "__main__":
+    # run the shell command `python antareslauncher/__init__.py ` to check
+    _check_metadata()

antareslauncher/advanced_launch.py

@@ -1,23 +1,17 @@
 from pathlib import Path
 
-from antareslauncher.main import run_with, MainParameters
-from antareslauncher.main_option_parser import (
-    MainOptionParser,
-    ParserParameters,
-)
+from antareslauncher.config import Config, get_config_path
+from antareslauncher.main import MainParameters, run_with
+from antareslauncher.main_option_parser import MainOptionParser, ParserParameters
 from antareslauncher.parameters_reader import ParametersReader
 
-HERE = Path(__file__).parent.resolve()
-PACKAGE_NAME = "antareslauncher"
-PROJECT_DIR = next(iter(p for p in HERE.parents if p.joinpath(PACKAGE_NAME).exists()))
-DATA_DIR = PROJECT_DIR / "data"
-SSH_JSON_FILE = DATA_DIR / "ssh_config.json"
-YAML_CONF_FILE = DATA_DIR / "configuration.yaml"
-
 
 def main():
+    config_path: Path = get_config_path()
+    config = Config.load_config(config_path)
     param_reader = ParametersReader(
-        json_ssh_conf=SSH_JSON_FILE, yaml_filepath=YAML_CONF_FILE
+        json_ssh_conf=config.ssh_config.config_path,
+        yaml_filepath=config.config_path,
     )
     parser_parameters: ParserParameters = param_reader.get_parser_parameters()
     parser: MainOptionParser = MainOptionParser(parser_parameters)

antareslauncher/basic_launch.py

@@ -1,21 +1,17 @@
-import sys
 from pathlib import Path
 
-from antareslauncher.main import run_with, MainParameters
-from antareslauncher.main_option_parser import (
-    MainOptionParser,
-    ParserParameters,
-)
+from antareslauncher.config import Config, get_config_path
+from antareslauncher.main import MainParameters, run_with
+from antareslauncher.main_option_parser import MainOptionParser, ParserParameters
 from antareslauncher.parameters_reader import ParametersReader
 
-DATA_DIR = Path(__file__).parent.resolve() / "data"
-SSH_JSON_FILE = DATA_DIR / "ssh_config.json"
-YAML_CONF_FILE = DATA_DIR / "configuration.yaml"
-
 
 def main():
+    config_path: Path = get_config_path()
+    config = Config.load_config(config_path)
     param_reader = ParametersReader(
-        json_ssh_conf=SSH_JSON_FILE, yaml_filepath=YAML_CONF_FILE
+        json_ssh_conf=config.ssh_config.config_path,
+        yaml_filepath=config.config_path,
     )
     parser_parameters: ParserParameters = param_reader.get_parser_parameters()
     parser: MainOptionParser = MainOptionParser(parser_parameters)