Skip to content

Commit

Permalink
Merge pull request #24 from willu47/will/environments
Browse files Browse the repository at this point in the history
Review of workflow
  • Loading branch information
willu47 authored Aug 29, 2022
2 parents 6cfbf93 + dd1d67a commit 26d3f71
Show file tree
Hide file tree
Showing 1,516 changed files with 1,840 additions and 103 deletions.
Binary file removed .DS_Store
Binary file not shown.
55 changes: 55 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Temporary and binary files
*~
*.py[cod]
*.so
*.cfg
!.isort.cfg
!setup.cfg
*.orig
*.log
*.pot
__pycache__/*
.cache/*
.*.swp
*/.ipynb_checkpoints/*
.DS_Store
.ipynb_checkpoints/
.snakemake

# Project files
.ropeproject
.project
.pydevproject
.settings
.idea
tags
.vscode

# Package files
*.egg
*.eggs/
.installed.cfg
*.egg-info

# Unittest and coverage
htmlcov/*
.coverage
.tox
junit.xml
coverage.xml
.pytest_cache/

# Build and docs folder/files

# Per-project virtualenvs
.venv*/

# Plots
*.pdf

# Temporary workflow files
working_directory/*

# Outputs
results/*
*.png
49 changes: 29 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,35 +1,44 @@
# OSeMBE_ECEMF
This repository contains the scenarios modelled in OSeMBE throughout the H2020 project ECEMF -European Climate and Energy Modelling Forum. And the repo contains a workflow to run OSeMOSYS models from datapackage to results file in IAMC format.
# OSeMOSYS workflow
This workflow allows to run one or multiple scenarios. Starting from an OSeMOSYS datapackage going through all steps running the model with gurobi and producing the results in IAMC format.

In addition to the standard OSeMOSYS outputs, the workflow can produce csvs containing the dual values for all constrains in the model.
This repository contains the scenarios modelled in OSeMBE throughout the H2020 project ECEMF -
European Climate and Energy Modelling Forum.

The repo contains a workflow to run OSeMOSYS models from datapackage to results file in IAMC format.

## OSeMOSYS workflow

This workflow allows to run one or multiple scenarios.
Starting from an OSeMOSYS datapackage going through all steps,
running the model with the Gurobi solver and producing the results in IAMC format.

In addition to the standard OSeMOSYS outputs, the workflow can produce CSV files containing the dual values for all constraints in the model.

### Installation

## Installation
Install snakemake using conda into a new environment called `snakemake`:

```bash
conda install -c conda-forge mamba
mamba create -c bioconda -c conda-forge -n snakemake snakemake-minimal
```
## Configuring the workflow
### Input:

- datapackage(s)
The workflow manages dependencies through conda environments.
Dependencies are defined per rule and are installed upon first running the workflow.

### Output:
### Configuring the workflow

- xlsx-files with model results in IAMC format
3. Place the script `resultify.py` from the repo [osemosys2iamc](https://github.com/OSeMOSYS/osemosys2iamc/tree/osembe) in the root folder of the project.

### Usage:
### Adding new scenarios

1. Place datapackage(s) in the folder `input_data`. Each datapackage should be placed in a folder
that is named after the scenario, e.g. `baseline`.
4. Check that the file `config.yaml`, which defines the conversion of OSeMOSYS results to IAMC format
is suitable for the model.

### Running the workflow

1. Place datapackage(s) in the folder `data`. Each datapackage should be in a folder that is named after the scenario, e.g. `baseline`.
2. Open the snakefile in a text-editor of your choice. Replace the scenario names potentially already in the list of `SCENARIOS` in line 1 with the names of your scenarios. The line should look similar to the below:
```bash
SCENARIOS = ['baseline', 'scenario1']`
```
3. Place the script `resultify.py` from the repo [osemosys2iamc](https://github.com/OSeMOSYS/osemosys2iamc/tree/osembe) in your folder.
4. Make sure that the file `data/config.yaml`is suitable for your model.
3. ***Optional***: To retrieve dual values from your model you need to edit the list of `constraints` in the file `run.py`.
4. Open terminal or command prompt in the directory of the snakefile.
4. Open terminal or command prompt and change to the directory of the snakefile.
5. ***Optional***: Perform a dry run to test snakemake with the command: `snakemake -n`
5. Start the scenario runs with the command `snakemake --cores <number of cores to be used> --use-conda`
5. Start the scenario runs with the command `snakemake --cores <number of cores to be used> --use-conda`
124 changes: 124 additions & 0 deletions config/dp_files.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
data/ATReserveMarginTagTechnology.csv
data/AccumulatedAnnualDemand.csv
data/AnnualEmissionLimit.csv
data/AnnualExogenousEmission.csv
data/AvailabilityFactor.csv
data/BEReserveMarginTagTechnology.csv
data/BGReserveMarginTagTechnology.csv
data/CHReserveMarginTagTechnology.csv
data/CYReserveMarginTagTechnology.csv
data/CZReserveMarginTagTechnology.csv
data/CapacityFactor.csv
data/CapacityOfOneTechnologyUnit.csv
data/CapacityToActivityUnit.csv
data/CapitalCost.csv
data/CapitalCostStorage.csv
data/Conversionld.csv
data/Conversionlh.csv
data/Conversionls.csv
data/DAILYTIMEBRACKET.csv
data/DAYTYPE.csv
data/DEReserveMarginTagTechnology.csv
data/DKReserveMarginTagTechnology.csv
data/DaySplit.csv
data/DaysInDayType.csv
data/DepreciationMethod.csv
data/DiscountRate.csv
data/DiscountRateStorage.csv
data/EEReserveMarginTagTechnology.csv
data/EMISSION.csv
data/ESReserveMarginTagTechnology.csv
data/EmissionActivityRatio.csv
data/EmissionsPenalty.csv
data/FIReserveMarginTagTechnology.csv
data/FRReserveMarginTagTechnology.csv
data/FUEL.csv
data/FixedCost.csv
data/GRReserveMarginTagTechnology.csv
data/HRReserveMarginTagTechnology.csv
data/HUReserveMarginTagTechnology.csv
data/IEReserveMarginTagTechnology.csv
data/ITReserveMarginTagTechnology.csv
data/InputActivityRatio.csv
data/LTReserveMarginTagTechnology.csv
data/LUReserveMarginTagTechnology.csv
data/LVReserveMarginTagTechnology.csv
data/MODE_OF_OPERATION.csv
data/MTReserveMarginTagTechnology.csv
data/MinStorageCharge.csv
data/ModelPeriodEmissionLimit.csv
data/ModelPeriodExogenousEmission.csv
data/NLReserveMarginTagTechnology.csv
data/NOReserveMarginTagTechnology.csv
data/OperationalLife.csv
data/OperationalLifeStorage.csv
data/OutputActivityRatio.csv
data/PLReserveMarginTagTechnology.csv
data/PTReserveMarginTagTechnology.csv
data/REGION.csv
data/REMinProductionTarget.csv
data/RETagFuel.csv
data/RETagTechnology.csv
data/ROReserveMarginTagTechnology.csv
data/ReserveMargin.csv
data/ReserveMarginTagFuel.csv
data/ResidualCapacity.csv
data/ResidualStorageCapacity.csv
data/SEASON.csv
data/SEReserveMarginTagTechnology.csv
data/SIReserveMarginTagTechnology.csv
data/SKReserveMarginTagTechnology.csv
data/STORAGE.csv
data/SpecifiedAnnualDemand.csv
data/SpecifiedDemandProfile.csv
data/StorageLevelStart.csv
data/StorageMaxChargeRate.csv
data/StorageMaxDischargeRate.csv
data/TECHNOLOGY.csv
data/TIMESLICE.csv
data/TechnologyFromStorage.csv
data/TechnologyToStorage.csv
data/TotalAnnualMaxCapacity.csv
data/TotalAnnualMaxCapacityInvestment.csv
data/TotalAnnualMinCapacity.csv
data/TotalAnnualMinCapacityInvestment.csv
data/TotalTechnologyAnnualActivityLowerLimit.csv
data/TotalTechnologyAnnualActivityUpperLimit.csv
data/TotalTechnologyModelPeriodActivityLowerLimit.csv
data/TotalTechnologyModelPeriodActivityUpperLimit.csv
data/TradeRoute.csv
data/UKReserveMarginTagTechnology.csv
data/VariableCost.csv
data/YEAR.csv
data/YearSplit.csv
data/YearlyPeakAT.csv
data/YearlyPeakBE.csv
data/YearlyPeakBG.csv
data/YearlyPeakCH.csv
data/YearlyPeakCY.csv
data/YearlyPeakCZ.csv
data/YearlyPeakDE.csv
data/YearlyPeakDK.csv
data/YearlyPeakEE.csv
data/YearlyPeakES.csv
data/YearlyPeakFI.csv
data/YearlyPeakFR.csv
data/YearlyPeakGR.csv
data/YearlyPeakHR.csv
data/YearlyPeakHU.csv
data/YearlyPeakIE.csv
data/YearlyPeakIT.csv
data/YearlyPeakLT.csv
data/YearlyPeakLU.csv
data/YearlyPeakLV.csv
data/YearlyPeakMT.csv
data/YearlyPeakNL.csv
data/YearlyPeakNO.csv
data/YearlyPeakPL.csv
data/YearlyPeakPT.csv
data/YearlyPeakRO.csv
data/YearlyPeakSE.csv
data/YearlyPeakSI.csv
data/YearlyPeakSK.csv
data/YearlyPeakUK.csv
data/default_values.csv
4 changes: 4 additions & 0 deletions envs/dag.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
channels:
- anaconda
dependencies:
- graphviz
2 changes: 1 addition & 1 deletion envs/openentrance_env.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,5 @@ dependencies:
- pip:
- pyam-iamc>=1.0
- iam-units>=2021.11.12
- git+git://github.com/IAMconsortium/nomenclature.git
- nomenclature-iamc
- git+https://github.com/openENTRANCE/openentrance.git@main#egg=openentrance
2 changes: 1 addition & 1 deletion envs/otoole_env.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,4 @@ dependencies:
- xlrd
- pip
- pip:
- otoole
- git+https://github.com/HauHe/otoole.git@osembe#egg=otoole
31 changes: 0 additions & 31 deletions gen_lp.py

This file was deleted.

File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Loading

0 comments on commit 26d3f71

Please sign in to comment.