Add section 'requirements for processing modules'

scenario-databases.rst

@@ -76,17 +76,22 @@ Scenario processing
 -------------------
 
 When submitting a scenario (a.k.a. "run") to an IIASA database instance, the server
-executes a scenario-processing workflow including *region-aggregation* and
-*scenario validation* prior to saving the scenario to the database. The processing uses
-the **nomenclature** package (`read the docs <https://nomenclature-iamc.readthedocs.io>`_).
+executes a scenario-processing workflow including *region-aggregation* and *scenario
+validation* prior to saving the scenario to the database. The processing uses the
+**nomenclature** package (`read the docs <https://nomenclature-iamc.readthedocs.io>`_).
 
 The region-aggregation and validation is configured via a project-specific GitHub_
 repository, usually named `https://github.com/iiasa/<project>-workflow`_. Please contact
 the respective project managers or the Scenario Services team if you need access.
 
 You can also run the project workflow locally (on your computer) before submission to
 an IIASA database instance, to make sure that the validation and processing works.
-See :ref:`local-processing` for more information!
+See :ref:`local-processing` for more information.
+
+The workflow for processing files uploaded via the IIASA Scenario Explorer is
+implemented in a modular fashion. It is possible to execute programs, code and tools
+developed by (non-IIASA) research partners as part of the processing workflow
+if the tool follows the :ref:`processing-requirements`.
 
 .. _GitHub: https://www.github.com
 

user-guide.rst

@@ -33,8 +33,9 @@ Detailed User Guides
 --------------------
 
 .. toctree::
-   :maxdepth: 2
+   :maxdepth: 1
 
    user-guide/local-processing
+   user-guide/processing-requirements
 
 .. _common-definitions: https://github.com/iamconsortium/common-definitions

user-guide/processing-requirements.rst

@@ -0,0 +1,54 @@
+.. _processing-requirements:
+
+Requirements for processing modules
+===================================
+
+Any module (a.k.a. program, code or tool) must adhere to the following standards of
+best-practice software development. The aim of these guidelines is to ensure reliability
+of our services, minimize maintenance requirements, and guarantee reproducibility of
+results across platforms.
+
+General requirements
+--------------------
+
+- The program, code or tool must be implemented in Python (≥3.10) or R; compiled
+  executables are not acceptable for security reasons
+- Distribution of the source code
+  - via an online version-controlled repository
+  (preferably GitHub) to which the IIASA admin team has access; or
+  - installation via a package manager (pip, conda, CRAN).
+- The program must run on Debian (preferably Ubuntu)
+- The dependencies must be clearly stated,
+  e.g. as Dockerfile (describing execution environment, library dependencies etc.)
+  Python package dependencies according to packaging user guide (e.g. as environment.yml, requirements.txt etc.)
+  R dependencies
+- The license must be clearly stated.
+- The documentation of the program, code or tool must include:
+  - Purpose of the program and individual top-level functions
+  - Instructions how to run the program
+  - Expected input (variables, region mappings) and standard output
+  - Explanation of any settings and optional parameters
+
+Application programming interface
+---------------------------------
+
+**Option 1**:
+
+The module is called via a command-line interface (CLI)
+and take the following arguments:
+
+- :code:`input`: path to an IAMC-formatted file (:code:`xlsx` or :code:`csv`)
+- :code:`output`: path where to write an output file
+  (usually derived timeseries data) in the same format
+- Any relevant settings and optional parameters must also be specified
+  via the CLI
+
+e.g. :code:`"python process.py --input path-to-input-file.xlsx --output path-to-output-file.xlsx"`
+
+**Option 2** (applicable for packages/functions written in Python):
+
+Importable Python functions that take and return :class:`pandas.DataFrame` (with columns
+folllowing the IAMC format) or :class:`pyam.IamDataFrame` objects can be called as part
+of the processing workflow. Any settings or optional parameters must be given as keyword
+arguments to the top-level function, preferably with the option to set them via a
+settings or configuration file.