Release candidate 1.2 (#586)

* Add test-publish workflow

* Update the release procedure

* Add publish-on-pypi workflow

* Try removing versioneer

* remove get_version() from __init__.py

* Move setup-config to cfg

* Fix linebreak in requirements list and formatting

* Fix more linebreaks in requirements

* Add package-directory directive

* Add explanations to the "publish"-workflow, remove test-publish workflow

* Update the release procedure some more

* Add a more elegant way to get the version number

* Add highlights to release notes

* Update the installation page

* Make black

* Python 3.7 doesn't support extras-require in setup.cfg with `-`

* Remove argparse dependency

* Shorten the badges for the listserv

* Fix package name

* Try alternative approach for version number

* Try adding importlib-metadata explicitly

* Try adding importlib explicitly

* Yet another try to read the version number

* Appease stickler

* Add classifiers and documentation url

* Add logos to the docs folder

* Implement changes requested by @gidden

* Add explicit item numbers in the release procedure, minor edits

* Harmonize workflow names

* Update R-tutorial to include reading data from an IIASA data resource

.github/workflows/build-docs.yml

@@ -36,7 +36,7 @@ jobs:
       uses: r-lib/actions/setup-pandoc@v1
 
     - name: Install dependencies and package for building the docs
-      run: pip install -e .[tutorials,optional-plotting,docs]
+      run: pip install -e .[tutorials,optional_plotting,docs]
 
     - name: Build the docs
       run: make --directory=doc html

.github/workflows/nightly.yml

@@ -40,7 +40,7 @@ jobs:
     - name: Install dependencies and package
       run: |
         python -m pip install --upgrade pip
-        pip install -e .[tests,deploy,optional-plotting,optional-io-formats,tutorials,docs]
+        pip install -e .[tests,deploy,optional_plotting,optional_io_formats,tutorials,docs]
 
     - name: Test with pytest (including Matplotlib)
       run: |

.github/workflows/publish.yml

@@ -0,0 +1,50 @@
+name: publish
+
+on:
+  push:
+    # build the package on any push to a release-candidate branch
+    branches: [ "release/rc_v*" ]
+    # deploy to test-pypi on any push of a version tag
+    tags: [ "v*" ]
+  release:
+    # deploy to pypi when publishing a new release
+    types: [ published ]
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v2
+
+    - uses: actions/setup-python@v2
+
+    - name: Cache Python packages
+      uses: actions/cache@v2
+      with:
+        path: |
+          ~/.cache/pip
+        key: publish-${{ runner.os }}
+
+    - name: Upgrade pip, wheel, setuptools-scm
+      run: python -m pip install --upgrade pip wheel setuptools-scm twine
+
+    - name: Build package
+      run: |
+        python3 setup.py bdist_wheel sdist
+        twine check dist/*
+
+    - name: Publish to TestPyPI
+      uses: pypa/[email protected]
+      if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags')
+      with:
+        user: __token__
+        password: ${{ secrets.TESTPYPI_TOKEN }}
+        repository_url: https://test.pypi.org/legacy/
+
+    - name: Publish to PyPI
+      uses: pypa/[email protected]
+      if: github.event_name == 'release'
+      with:
+        user: __token__
+        password: ${{ secrets.PYPI_TOKEN }}

.github/workflows/pytest-dependency.yml

@@ -28,7 +28,7 @@ jobs:
       run: pip install pandas==1.1.1 numpy==1.19.0 matplotlib==3.2.0 iam-units==2020.4.21 xlrd==2.0
 
     - name: Install other dependencies and package
-      run: pip install -e .[tests,deploy,optional-plotting,optional-io-formats,tutorials]
+      run: pip install -e .[tests,deploy,optional_plotting,optional_io_formats,tutorials]
 
     - name: Test with pytest
       run: pytest tests --mpl

.github/workflows/pytest.yml

@@ -20,6 +20,7 @@ jobs:
         python-version:
         - '3.9'
         - '3.8'
+        # Remove dependency `importlib_metadata` when dropping support for Python 3.7
         - '3.7'
 
       fail-fast: false
@@ -36,7 +37,7 @@ jobs:
         python-version: ${{ matrix.python-version }}
 
     - name: Install dependencies and package
-      run: pip install -e .[tests,deploy,optional-plotting,optional-io-formats,tutorials]
+      run: pip install -e .[tests,deploy,optional_plotting,optional_io_formats,tutorials]
 
     - name: Test with pytest
       if: ${{ matrix.python-version != '3.9' }}

Makefile

@@ -77,7 +77,7 @@ virtual-environment: $(VENV_DIR)  ## make virtual environment for development
 
 $(VENV_DIR):
 	$(VENV_DIR)/bin/pip install --upgrade pip
-	$(VENV_DIR)/bin/pip install -e .[tests,optional-io-formats]
+	$(VENV_DIR)/bin/pip install -e .[tests,optional_io_formats]
 	$(CONDA_EXE) env update --name $(CONDA_DEFAULT_ENV) --file $(DOC_ENVIRONMENT_CONDA_FILE)
 
 	touch $(VENV_DIR)

README.md

@@ -16,7 +16,7 @@ pyam: analysis & visualization <br /> of integrated-assessment and macro-energy
 [![doi](https://zenodo.org/badge/113359260.svg)](https://doi.org/10.5281/zenodo.1470400)
 [![ORE](https://img.shields.io/badge/ORE-10.12688/openreseurope.13633.2-blue)](https://doi.org/10.12688/openreseurope.13633.2)
 [![joss](https://joss.theoj.org/papers/10.21105/joss.01095/status.svg)](https://joss.theoj.org/papers/10.21105/joss.01095)
-[![groups.io](https://img.shields.io/badge/listserv-groups.io-blue)](https://pyam.groups.io/g/forum)
+[![groups.io](https://img.shields.io/badge/mail-groups.io-blue)](https://pyam.groups.io/g/forum)
 [![slack](https://img.shields.io/badge/chat-Slack-orange)](https://pyam-iamc.slack.com)
 
 ****
@@ -149,7 +149,7 @@ To install from source (including all dependencies)
 after cloning this repository, simply run
 
 ```
-pip install --editable .[tests,optional-io-formats,tutorials]
+pip install --editable .[tests,optional_io_formats,tutorials]
 ```
 
 To check that the package was installed correctly, run

RELEASE_NOTES.md

@@ -1,8 +1,16 @@
-# Next Release
+# Release v1.2.0
+
+## Highlights
+
+- Update the source code of the manuscript in *Open Research Europe* to reflect changes
+  based on reviewer comments
+- Increase the performance of the IamDataFrame initialization
+- Add an experimental "profiler" module for performance benchmarking
 
 ## Dependency changes
 
-The dependencies were updated to require `xlrd>=2.0` (previously `<2.0`) and `openpyxl` was added as a dependency.
+The dependencies were updated to require `xlrd>=2.0` (previously `<2.0`) and `openpyxl`
+was added as a dependency.
 
 ## Individual updates
 

RELEASE_PROCEDURE.md

@@ -1,4 +1,3 @@
-
 # Release procedure
 
 ## Required accounts and admin privileges
@@ -9,48 +8,52 @@
 
 ## Steps for publishing a new release
 
-1. Make a release candidate branch (e.g., `release/rc_v<release version>`)
+1. Create a release candidate branch named `release/rc_v<release version>`
    and pull request it into `main` with the following updates:
    1. If it's the first release in a new year,
-      search for `Copyright 2017` and bump the year
-   1. Deprecate any stated portion of the API
+      search for `Copyright 2017` and update the end-year of the copyright tag
+   2. Deprecate any portion of the API marked for removal in this release
       (you can find them by searching the code base for "deprecate")
-   1. Update `RELEASE_NOTES.md` (see the examples of previous releases)
-	  - replace "# Next Release" with "# Release v<release version>"
-	  - add a "## Highlights" section with the most important updates & changes
-      - if applicable, add/review "## Dependency changes" and "## API changes" sections 
-      - add a new heading "## Individual Updates" before the PR listing
-  1. Confirm that the PR passes all tests and checks
-  1. Tag the release number: `git tag v<release version>`, e.g., `git tag v1.2.0`
-     - **THIS IS NOT THE TAGGED COMMIT WE WILL DISTRIBUTE, IT IS ONLY FOR TESTING**
-	 - **DO NOT PUSH THIS TAG TO UPSTREAM**
-  1. Run `make publish-on-testpypi`
-     - this should "just work" - if it does not, fix any issues,
-       retag (`git tag -d` then `git tag`), and try again
-  1. Once successful, delete the tag, and merge the candidate PR into `main` on Github
-1. Switch to the now-updated main branch: `git checkout main` and `git pull upstream main`
-1. Tag the release number: `git tag v<release version>`, e.g., `git tag v1.2.0`
-   - `versioneer` automatically updates the version number based on the tag
-   - this is now the official tagged commit
-1. Push the tag upstream: `git push upstream --tags`
-1. Run `make publish-on-pypi`
-   - this will make wheels that allow `pyam` to be installed via `pip install`
-   - check that the new version is available at https://pypi.org/project/pyam-iamc/
-1. Make a new release on Github
-   - make sure that you choose the tag name defined above
-   - copy the release summary from `RELEASE_NOTES.md` into the description box
-1. Update on `conda-forge`
-   - a PR should automatically be opened by the bot after the Github release
-   - confirm that any new depedencies are included,
+   3. Update `RELEASE_NOTES.md` (see the examples of previous releases)
+	  - Replace "# Next Release" with "# Release v<release version>"
+	  - Add a "## Highlights" section with the most important updates & changes
+      - If applicable, add/review "## Dependency changes" and "## API changes" sections 
+      - Add a new heading "## Individual Updates" before the list of individual PRs
+   4. Confirm that the PR passes all tests
+   5. Tag the release candidate `git tag v<release version>rc<n>`,
+      e.g., `git tag v1.2.0rc1`, and push to the upstream repository
+   6. Confirm that the "publish" workflow passes 
+      https://github.com/IAMconsortium/pyam/actions/workflows/publish.yml
+   7. Confirm that the release is published on https://test.pypi.org/project/pyam-iamc/
+      - The package can be downloaded, installed and run
+      - The README is rendered correctly
+   8. If there are any problems, fix the issues and repeat from step 5,
+      bumping the release candidate number  
+   9. If successful, merge the candidate PR into `main` and then delete the branch
+2. Switch to the updated main branch: `git checkout main` and `git pull upstream main`
+3. Tag the release number: `git tag v<release version>`, e.g., `git tag v1.2.0`
+4. Push the tag upstream: `git push upstream --tags`
+5. Make a new release on Github
+   - Make sure that you choose the tag name defined above
+   - Copy the release summary from `RELEASE_NOTES.md` into the description box
+6. Confirm that the "publish" workflow passes 
+   https://github.com/IAMconsortium/pyam/actions/workflows/publish.yml
+7. Confirm that the release is published on https://www.pypi.org/project/pyam-iamc/
+8. Update on `conda-forge`
+   - A PR should automatically be opened by the bot after the Github release
+   - Confirm that any new depedencies are included,
      change the minimum dependency version if necessary
      (compare to ./.github/workflows/pytest-depedency.yml)
-   - merge the PR
-   - check that the new version is available at https://anaconda.org/conda-forge/pyam
-1. Confirm that the doc pages are updated on https://pyam-iamc.readthedocs.io/
-   - both the **latest** and the **stable** versions point to the new release
-   - the new release has been added to the list of available versions
-1. Add a new line "# Next Release" at the top of `RELEASE_NOTES.md` and commit to `main`
-1. Announce it on our mailing list: https://groups.io/g/pyam
-   - again, copy the rendered HTML from the Github release directly in the email
+   - Merge the PR
+   - Check that the new version is available on https://anaconda.org/conda-forge/pyam
+9. Confirm that the doc pages are updated on https://pyam-iamc.readthedocs.io/
+   - Both the **latest** and the **stable** versions point to the new release
+   - The new release has been added to the list of available versions
+10. Add a new first line "# Next Release" in `RELEASE_NOTES.md` and commit to `main`
+11. Announce it to our community
+    - The mailing list (https://pyam.groups.io) - copy the rendered HTML
+      from the Github release and use the subject line `#release v<release version>`
+    - The Slack channel
+    - Social media using the tag `#pyam_iamc` 
 
 And that's it! Whew...

doc/source/R_tutorials/pyam_R_tutorial.ipynb

@@ -148,7 +148,7 @@
     {
      "data": {
       "text/html": [
-       "<table>\n",
+       "<table class=\"dataframe\">\n",
        "<caption>A data.frame: 3 × 7</caption>\n",
        "<thead>\n",
        "\t<tr><th scope=col>model</th><th scope=col>scenario</th><th scope=col>region</th><th scope=col>variable</th><th scope=col>unit</th><th scope=col>X2005</th><th scope=col>X2010</th></tr>\n",
@@ -232,7 +232,7 @@
      "data": {
       "text/plain": [
        "<class 'pyam.core.IamDataFrame'>\n",
-       "Index dimensions:\n",
+       "Index:\n",
        " * model    : model_a (1)\n",
        " * scenario : scen_a, scen_b (2)\n",
        "Timeseries data coordinates:\n",
@@ -274,7 +274,7 @@
     {
      "data": {
       "text/html": [
-       "<table>\n",
+       "<table class=\"dataframe\">\n",
        "<caption>A data.frame: 6 × 7</caption>\n",
        "<thead>\n",
        "\t<tr><th scope=col>model</th><th scope=col>scenario</th><th scope=col>region</th><th scope=col>variable</th><th scope=col>unit</th><th scope=col>year</th><th scope=col>value</th></tr>\n",
@@ -374,7 +374,7 @@
     {
      "data": {
       "text/html": [
-       "<table>\n",
+       "<table class=\"dataframe\">\n",
        "<caption>A data.frame: 9 × 7</caption>\n",
        "<thead>\n",
        "\t<tr><th scope=col>model</th><th scope=col>scenario</th><th scope=col>region</th><th scope=col>variable</th><th scope=col>unit</th><th scope=col>year</th><th scope=col>value</th></tr>\n",
@@ -448,6 +448,81 @@
     "df$data"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "d4fa2637",
+   "metadata": {},
+   "source": [
+    "## Query data from an IIASA data resource\n",
+    "\n",
+    "The IIASA *Energy, Climate, and Environment* Program hosts a suite of **Scenario Explorer** instances and related infrastructure to support analysis of integrated-assessment pathways in IPCC reports and model comparison projects. \n",
+    "High-profile use cases include the [IAMC 1.5°C Scenario Explorer hosted by IIASA](https://data.ene.iiasa.ac.at/iamc-1.5c-explorer) supporting the *IPCC Special Report on Global Warming of 1.5°C* (SR15) and the Horizon 2020 project [CD-LINKS](https://data.ene.iiasa.ac.at/cd-links).\n",
+    "\n",
+    "The **pyam** package can retrieve scenario data directly from any **Scenario Explorer** database instance\n",
+    "hosted by IIASA for use in your processing and analysis workflows."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "91c7fddf",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df = pyam$read_iiasa(\n",
+    "    'iamc15',\n",
+    "    model='MESSAGEix*', \n",
+    "    variable=c('Emissions|CO2', 'Primary Energy|Coal'), \n",
+    "    region='World',\n",
+    "    meta=c('category')\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "377eac4c",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "<class 'pyam.core.IamDataFrame'>\n",
+       "Index:\n",
+       " * model    : MESSAGEix-GLOBIOM 1.0 (1)\n",
+       " * scenario : CD-LINKS_INDCi, CD-LINKS_NPi, CD-LINKS_NPi2020_1000, ... LowEnergyDemand (7)\n",
+       "Timeseries data coordinates:\n",
+       "   region   : World (1)\n",
+       "   variable : Emissions|CO2, Primary Energy|Coal (2)\n",
+       "   unit     : EJ/yr, Mt CO2/yr (2)\n",
+       "   year     : 2000, 2005, 2010, 2020, 2030, 2040, 2050, 2060, ... 2100 (12)\n",
+       "Meta indicators:\n",
+       "   version (object) 1 (1)\n",
+       "   carbon price|Avg NPV (2030-2100) (object) 0.9123249940000001, ... 30.1503013256 (7)\n",
+       "   year of netzero CO2 emissions (object) nan, 2065, 2078, 2053, 2060.0 (5)\n",
+       "   carbon price|2100 (NPV) (object) 0.0426419715, 0.012143570900000001, ... 14.2051784281 (7)\n",
+       "   cumulative CO2 emissions (2016 to peak warming, Gt CO2) (object) ... (7)\n",
+       "   ..."
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "df"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a0eb9b43",
+   "metadata": {},
+   "source": [
+    "See the [pyam-IIASA-database tutorial](https://pyam-iamc.readthedocs.io/en/stable/tutorials/iiasa_dbs.html)\n",
+    "or the [API documentation](https://pyam-iamc.readthedocs.io/en/stable/api/iiasa.html)\n",
+    "for more information and a complete list of features!"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "isolated-whole",

doc/source/index.rst

@@ -46,7 +46,7 @@ Release v\ |version|.
 .. |joss| image:: https://joss.theoj.org/papers/10.21105/joss.01095/status.svg
    :target: https://joss.theoj.org/papers/10.21105/joss.01095
 
-.. |groupsio| image:: https://img.shields.io/badge/listserv-groups.io-blue
+.. |groupsio| image:: https://img.shields.io/badge/mail-groups.io-blue
    :target: https://pyam.groups.io/g/forum
 
 .. |slack| image:: https://img.shields.io/badge/chat-Slack-orange.svg

doc/source/install.rst

@@ -20,6 +20,7 @@ Pip
 
     pip install pyam-iamc
 
+.. warning::  The pyam package is distributed as "pyam-iamc" on pypi.
 
 Installing From Source
 ----------------------
@@ -34,10 +35,8 @@ Dependencies
 ------------
 
 Like any software project, we stand on the shoulders of giants. Our particular
-giants include **numpy** :cite:`numpy`, **matplotlib**
-:cite:`matplotlib`, and **pandas** :cite:`pandas`. Explicit requirements are
-fully enumerated below.
+giants include **pandas** :cite:`pandas`, **matplotlib** :cite:`matplotlib`,
+and **numpy** :cite:`numpy`.
+See the `setup-configuration`_ for more information.
 
-The required depedencies for |pyam| are:
-
-  .. program-output:: python -c 'import sys; sys.path.insert(0, "../.."); from setup import REQUIREMENTS; print("\n".join([r for r in REQUIREMENTS]))'
+.. _`setup-configuration`: https://github.com/IAMconsortium/pyam/blob/main/setup.cfg

doc/source/tutorials/iiasa_dbs.ipynb

@@ -6,7 +6,7 @@
    "source": [
     "# Read directly from IIASA data resources\n",
     "\n",
-    "The IIASA Energy Program hosts a suite of **Scenario Explorer** instances and related infrastructure to support analysis of integrated-assessment pathways in IPCC reports and model comparison projects. \n",
+    "The IIASA *Energy, Climate, and Environment* Program hosts a suite of **Scenario Explorer** instances and related infrastructure to support analysis of integrated-assessment pathways in IPCC reports and model comparison projects. \n",
     "High-profile use cases include the [IAMC 1.5°C Scenario Explorer hosted by IIASA](https://data.ene.iiasa.ac.at/iamc-1.5c-explorer) supporting the *IPCC Special Report on Global Warming of 1.5°C* (SR15) and the Horizon 2020 project [CD-LINKS](https://data.ene.iiasa.ac.at/cd-links).\n",
     "\n",
     "IIASA's [modeling platform infrastructure](http://software.ene.iiasa.ac.at/ixmp-server) and the Scenario Explorer UI is not only a great resource on its own, but it also allows the underlying datasets to be directly queried.\n",