Direct action on deployment (#34)

* Prototype implementation of direct action on deployment numbers in the env.py under 'reference_environment_direct_deployment' folder, by using the pycel package, which unfortunately has not implemented Excel's RRI() function yet. An alternative solution is to use https://www.excelpython.org/koala/, which has a bug that needs to be fixed manually vallettea/koala#241 by changing all its files. * Corrected env.py and added a new test script test_reference_environment_direct_deployment.py for the new env.py for direct deployment. Compiling of the 'Pathways to Net Zero.xlsx' only happens during environment creation by env = gym.make("reference_environment_direct_deployment:reference-environment-direct-deployment-v0"), which should take 1 minute or less. Before running the test_reference_environment_direct_deployment.py lines-by-lines, make sure the current directory is netzerotc/environment/, and place the 'Pathways to Net Zero - Simplified.xlsx' (can be downloaded from MS Teams/EXT-ATI/OREC-RANGL/General/Files) inside netzerotc/environment/compiled_workbook_objects/. Note: All RRI() functions/formulae in the original 'Pathways to Net Zero.xlsx' have been replaced by the original values, and 3 controllable technologies' all future deployment numbers from 2031 to 2050 have been removed because future is unknown in RL, and also, in the 'Outputs' spreadsheet of the 'Pathways to Net Zero.xlsx', all values depending on weighted sum of future values are changed to depending only on current year's values. The simplified/modified 'Pathways to Net Zero - Simplified.xlsx' contains changelog/history of changes so it's easy to see which cells' formulae have been replaced by raw values. * Fixed the compiled object's cell's map/address initialization issue: before resetting or setting any values in the compiled object, all reward components cells have to be evaluated first in order to get initialized, and after setting the action of deployment increments, the reward component cells have to be re-evaluated again to take the updated/reset value into consideration when doing the formula calculation. Now the env.py's output rewards components are (almost) exactly the same as Excel's workbook's total capex opex revenue emissions: In 'Pathways to Net Zero - Simplified.xlsx' - 'Gale' spreadsheet, simply apply an increments of [1.0 1.0 2.0] to offshore wind, blue hydrogen, and green hydrogen on top of 2030 values and set the new values as 2031 values, then apply constant increments [2.0 2.0 5.0] (as in the test_reference_environment_direct_deployment.py) for all following years, then in the 'Outputs' and 'CCUS', the total capex opex revenue and emissions from 2031 to 2050 are exactly the same as env.py's outputs. Moreover, even in the original 'Pathways to Net Zero.xlsx', when doing the same operation, the total opex revenue and emissions from 2031 to 2050 are still exactly the same as env.py's outputs, whereas the total capex is different because in the original 'Pathways to Net Zero.xlsx', offshore wind's capex depends on some numbers that are weighted sum of future values, which is not removed and replaced by current year's values only in the 'Pathways to Net Zero - Simplified.xlsx'. * Update README.md Update readme for actions to control deployment directly (rather than indirectly via scenarios) * Added requirement * Update README.md Update for direct deployment * Fix typos * Updated the jobs number calculation using a simple formula: 0.25 * (capex + opex + decomm) / 0.05. * The training works successfully now (but needs to switch between different folders before and after creating the environment by running env = gym.make("reference_environment_direct_deployment:reference-environment-direct-deployment-v0")). The training takes about 1 sec per episode, and the 10 uploaded modes are trained by 1000 episodes per model. Please feel free to do some evaluations using the codes in evaluate.py. * Fixed inconsistence in def to_observation(self): and def observation_space(self): such that the training and evaluation now work properly. Updated some trained models with 100 episodes per model and the plot. * Implemented the randomization on prices and costs in spreadsheet 'CCUS' row 23,24,26 and spreadsheet 'Outputs' row 148, 149, 150, 153, 154, 155, 158, 159, 163, 164, 165, 166; Trained 10 models with 100 episodes per model. * Fixed the evaluation of trained models. * Clipped the multiplicative noise by max(noise, 0.1) such that the price/cost will not become negative or zero. * Clipped noise by max(noise, 0.001) in env.py, re-trained the models and evalute them by plotting. * fix: apply black formatter to all python files * fix: use relative paths when loading resources * fix: typo in comment * test: add demonstration of serialisation/de-serialisation * Update README.md * Revert "fix: apply black formatter to all python files" This reverts commit 87f9d59. Successfully implemented pycel serialization (for cells involved in env.py, but not necessarily for other cells) by using the evaluate_all_and_serialize.py in 'compiled_workbook_objects', and changed env.py to de-serialize the workbook, and also env.reset will also reset the env.param by self.param = Parameters(). * Implemented #3 of #36 and #1 of #5. Training and evaluation tests passed, and there is no longer the data type or array dimension issues experienced before with the def to_observation(self): and the def observation_space(self):. * step: test containerisation * step: use nztc:5000 in place of environment:5000 * fix: use evalai_rangl network * step: add base evaluation content * Updated README.md for #34 (comment); Fixed indexing of 'obs_high' in def observation_space(self):. * Implemented #35 (comment) to reset param when `env.reset()` is invoked, as an alternative to the current approach of creating a completely new `self.param = Parameters()` which will take 0.2~0.25 seconds to load the serialized .pkl/.yml; Correctly initialize the `state.randomized_costs` by setting them to fixed (non-randomized) 2030's values; Increased the upper bounds of costs/prices in the observation space by 10 times; Included non-independently randomized costs/prices in the `state.randomized_costs` and the observation space for #5 (comment). * step: add README for random agent submission * Update README.md Edit title from 'Random-agent...' to 'Agent...' because this workflow is the same for all agents. Move testing up before submission to reflect first-time use case. Summarise sample terminal output. * Update README.md Replace the challenge overview content with a link to the content on EvalAI. Add a note about the submission folder. * Add control of noise in Parameters() in env.py; Record deployment numbers (after applying actions of increments) to state.deployments, and add plotting of deployment numbers from 2031 to 2050. Add compiling and serialization of IEV model workbook in evaluate_all_and_serialize.py. Co-authored-by: Jia-Chen Hua <[email protected]> Co-authored-by: moriartyjm <[email protected]> Co-authored-by: jia-chenhua <[email protected]> Co-authored-by: Tomasz Kosmala <[email protected]>
rangl-labs · Oct 15, 2021 · 32e321a · 32e321a
1 parent d3c9843
commit 32e321a
Show file tree

Hide file tree

Showing 106 changed files with 7,837 additions and 40 deletions.
diff --git a/README.md b/README.md
@@ -2,36 +2,7 @@
 
 Welcome to the RangL Pathways to Net Zero challenge repository! 
 
-To get started, read the challenge overview.
-
-## Challenge overview
-
-Your goal is to find the optimal pathway to net zero carbon emissions for the offshore industry.
-
-Three possible pathways named Breeze, Gale and Storm have been identified in the [Integrated Energy Vision](https://ore.catapult.org.uk/press-releases/reimagining-a-net-zero-north-sea-an-integrated-energy-vision-for-2050/) study by the Net Zero Technology Centre and the Offshore Renewables Catapult. 
-
-In this challenge you can explore the following variations on Breeze, Gale and Storm:
-
-* Implement a weighted combination of the three pathways, and
-* Control the rate at which each strategy is implemented.
-
-At the first timestep you choose the scenario weights (they should be non-negative, sum to 1, and should not change). At each timestep you can do this to Breeze, Gale or Storm:
-
-* Progress, by advancing 1 year
-* Accelerate, by advancing more than 1 year.
-
-Your action has the form ((weights),(years)). For example, the action (0, 0.5, 0.5, 1, 2, 1) corresponds to:
-
-* Placing weight 0 on Breeze, 0.5 on Gale and 0.5 on Storm
-* Advancing the Breeze schedule by 1 year, Gale by 2 years, and Storm by 1 year
-
-Note that:
-
-* The weights are fixed: i.e. the weights used at each timestep t>1 must be equal to the weights used at timestep 1
-* For each scenario and each step, the minimum number of years to advance is 1 
-* In the above example, accelerating Breeze would have no effect on the rewards since Breeze has weight 0
-
-Clearly, accelerating progress towards net zero reduces total carbon emissions. However it also tends to be more expensive, since technology costs tend to reduce over time. Your goal is to find the best balance.
+To get started, read the [challenge overview](http://51.132.63.168:8888/web/challenges/challenge-page/8/overview).
 
 ## The RangL environment
 
@@ -64,3 +35,7 @@ At each step, the RangL environment uses random noise to model real-world uncert
 ## Training 
 
 To get started with training RL agents, head to the `local_agent_training_and_evaluation` folder and check out the README.
+
+## Submission
+
+To submit your agent to the competition, head to the `random_agent_submission` folder and check out its README.
diff --git a/docker-compose.yml b/docker-compose.yml
@@ -0,0 +1,12 @@
+version: "3"
+services:
+  nztc:
+    build:
+      context: environment
+      dockerfile: Dockerfile
+    ports:
+      - 5000:5000
+
+networks:
+  default:
+    name: evalai_rangl
diff --git a/environment/10models_100episodes_DirectDeployment_MODEL_1.png b/environment/10models_100episodes_DirectDeployment_MODEL_1.png
diff --git a/environment/Dockerfile b/environment/Dockerfile
@@ -0,0 +1,11 @@
+FROM python:3.9-slim-buster
+
+WORKDIR /service
+COPY requirements.txt .
+RUN pip install -r requirements.txt
+
+COPY . /service
+RUN pip install .
+RUN pip list
+
+CMD ["python", "server.py"]
diff --git a/environment/MODEL_0_10models_100episodes_DirectDeploymentRandomized.png b/environment/MODEL_0_10models_100episodes_DirectDeploymentRandomized.png
diff --git a/...nt/MODEL_0_10models_100episodes_DirectDeploymentRandomized_max(noise,0.001).png b/...nt/MODEL_0_10models_100episodes_DirectDeploymentRandomized_max(noise,0.001).png
diff --git a/...ment/MODEL_0_10models_100episodes_DirectDeploymentRandomized_max(noise,0.1).png b/...ment/MODEL_0_10models_100episodes_DirectDeploymentRandomized_max(noise,0.1).png
diff --git a/environment/MODEL_1_10models_100episodes_DirectDeploymentRandomized.png b/environment/MODEL_1_10models_100episodes_DirectDeploymentRandomized.png
diff --git a/...nt/MODEL_1_10models_100episodes_DirectDeploymentRandomized_max(noise,0.001).png b/...nt/MODEL_1_10models_100episodes_DirectDeploymentRandomized_max(noise,0.001).png
diff --git a/...ment/MODEL_1_10models_100episodes_DirectDeploymentRandomized_max(noise,0.1).png b/...ment/MODEL_1_10models_100episodes_DirectDeploymentRandomized_max(noise,0.1).png
diff --git a/environment/MODEL_2_10models_100episodes_DirectDeploymentRandomized.png b/environment/MODEL_2_10models_100episodes_DirectDeploymentRandomized.png
diff --git a/...nt/MODEL_2_10models_100episodes_DirectDeploymentRandomized_max(noise,0.001).png b/...nt/MODEL_2_10models_100episodes_DirectDeploymentRandomized_max(noise,0.001).png
diff --git a/...ment/MODEL_2_10models_100episodes_DirectDeploymentRandomized_max(noise,0.1).png b/...ment/MODEL_2_10models_100episodes_DirectDeploymentRandomized_max(noise,0.1).png
diff --git a/environment/MODEL_3_10models_100episodes_DirectDeploymentRandomized.png b/environment/MODEL_3_10models_100episodes_DirectDeploymentRandomized.png
diff --git a/...nt/MODEL_3_10models_100episodes_DirectDeploymentRandomized_max(noise,0.001).png b/...nt/MODEL_3_10models_100episodes_DirectDeploymentRandomized_max(noise,0.001).png
diff --git a/...ment/MODEL_3_10models_100episodes_DirectDeploymentRandomized_max(noise,0.1).png b/...ment/MODEL_3_10models_100episodes_DirectDeploymentRandomized_max(noise,0.1).png
diff --git a/environment/MODEL_4_10models_100episodes_DirectDeploymentRandomized.png b/environment/MODEL_4_10models_100episodes_DirectDeploymentRandomized.png
diff --git a/...nt/MODEL_4_10models_100episodes_DirectDeploymentRandomized_max(noise,0.001).png b/...nt/MODEL_4_10models_100episodes_DirectDeploymentRandomized_max(noise,0.001).png
diff --git a/...ment/MODEL_4_10models_100episodes_DirectDeploymentRandomized_max(noise,0.1).png b/...ment/MODEL_4_10models_100episodes_DirectDeploymentRandomized_max(noise,0.1).png
diff --git a/environment/MODEL_5_10models_100episodes_DirectDeploymentRandomized.png b/environment/MODEL_5_10models_100episodes_DirectDeploymentRandomized.png
diff --git a/...nt/MODEL_5_10models_100episodes_DirectDeploymentRandomized_max(noise,0.001).png b/...nt/MODEL_5_10models_100episodes_DirectDeploymentRandomized_max(noise,0.001).png
diff --git a/...ment/MODEL_5_10models_100episodes_DirectDeploymentRandomized_max(noise,0.1).png b/...ment/MODEL_5_10models_100episodes_DirectDeploymentRandomized_max(noise,0.1).png
diff --git a/environment/MODEL_6_10models_100episodes_DirectDeploymentRandomized.png b/environment/MODEL_6_10models_100episodes_DirectDeploymentRandomized.png
diff --git a/...nt/MODEL_6_10models_100episodes_DirectDeploymentRandomized_max(noise,0.001).png b/...nt/MODEL_6_10models_100episodes_DirectDeploymentRandomized_max(noise,0.001).png
diff --git a/...ment/MODEL_6_10models_100episodes_DirectDeploymentRandomized_max(noise,0.1).png b/...ment/MODEL_6_10models_100episodes_DirectDeploymentRandomized_max(noise,0.1).png
diff --git a/environment/MODEL_7_10models_100episodes_DirectDeploymentRandomized.png b/environment/MODEL_7_10models_100episodes_DirectDeploymentRandomized.png
diff --git a/...nt/MODEL_7_10models_100episodes_DirectDeploymentRandomized_max(noise,0.001).png b/...nt/MODEL_7_10models_100episodes_DirectDeploymentRandomized_max(noise,0.001).png
diff --git a/...ment/MODEL_7_10models_100episodes_DirectDeploymentRandomized_max(noise,0.1).png b/...ment/MODEL_7_10models_100episodes_DirectDeploymentRandomized_max(noise,0.1).png
diff --git a/environment/MODEL_8_10models_100episodes_DirectDeploymentRandomized.png b/environment/MODEL_8_10models_100episodes_DirectDeploymentRandomized.png
diff --git a/...nt/MODEL_8_10models_100episodes_DirectDeploymentRandomized_max(noise,0.001).png b/...nt/MODEL_8_10models_100episodes_DirectDeploymentRandomized_max(noise,0.001).png
diff --git a/...ment/MODEL_8_10models_100episodes_DirectDeploymentRandomized_max(noise,0.1).png b/...ment/MODEL_8_10models_100episodes_DirectDeploymentRandomized_max(noise,0.1).png
diff --git a/environment/MODEL_9_10models_100episodes_DirectDeploymentRandomized.png b/environment/MODEL_9_10models_100episodes_DirectDeploymentRandomized.png
diff --git a/...nt/MODEL_9_10models_100episodes_DirectDeploymentRandomized_max(noise,0.001).png b/...nt/MODEL_9_10models_100episodes_DirectDeploymentRandomized_max(noise,0.001).png
diff --git a/...ment/MODEL_9_10models_100episodes_DirectDeploymentRandomized_max(noise,0.1).png b/...ment/MODEL_9_10models_100episodes_DirectDeploymentRandomized_max(noise,0.1).png
diff --git a/environment/Makefile b/environment/Makefile
@@ -0,0 +1,2 @@
+build:
+	docker build . -t environment:test
diff --git a/environment/README.md b/environment/README.md
@@ -1,8 +1,14 @@
 # RangL reference environment
 
+## Installation
+
+To install the environment, first copy a python object serialised from a suitable economic model spreadsheet with the filename `PathwaysToNetZero_Simplified_Compiled.pkl` into the `compiled_workbook_objects` folder.
+
+## Overview
+
 The `reference_environment` folder contains the environment used in the competition. To modify it for development purposes, check out its `README`.
 
-Modified environments can be checked using the `test_reference_environment.py` script. This will
+Modified environments can be checked using the `test_reference_environment_direct_deployment.py` script. This will
 * Run logical tests, which
     * validate that the environment is correctly specified
     * illustrate useful concepts such as random seeding

diff --git a/environment/compiled_workbook_objects/.gitignore b/environment/compiled_workbook_objects/.gitignore
@@ -0,0 +1,2 @@
+PathwaysToNetZero_Simplified_Compiled.pkl
+PathwaysToNetZero_Simplified_Compiled.yml
diff --git a/...ects/Pathways to Net Zero - Original - Fixed Policy Test - Rewards Component Extract.xlsx b/...ects/Pathways to Net Zero - Original - Fixed Policy Test - Rewards Component Extract.xlsx
diff --git a/environment/compiled_workbook_objects/Pathways to Net Zero - Simplified - Compiled.yml b/environment/compiled_workbook_objects/Pathways to Net Zero - Simplified - Compiled.yml
diff --git a/environment/compiled_workbook_objects/PathwaysToNetZero_Simplified_Compiled.yml b/environment/compiled_workbook_objects/PathwaysToNetZero_Simplified_Compiled.yml
diff --git a/environment/compiled_workbook_objects/evaluate_all_and_serialize.py b/environment/compiled_workbook_objects/evaluate_all_and_serialize.py
@@ -0,0 +1,63 @@
+# -*- coding: utf-8 -*-
+"""
+#Created on Fri Oct  8 2021
+
+@author: Administrator
+"""
+
+import numpy as np
+import time
+from pycel import ExcelCompiler
+
+Pathways2Net0 = ExcelCompiler("Pathways to Net Zero - Simplified.xlsx")
+
+Spreadsheets = np.array(['GALE!','CCUS!','Outputs!'])
+ColumnInds_BySheets = np.array([np.array(['P','X','Y']), 
+                                np.array(['O','P','Q','R','S','T','U','V','W','X','Y','Z','AA','AB','AC','AD','AE','AF','AG','AH','AI']), 
+                                np.array(['O','P','Q','R','S','T','U','V','W','X','Y','Z','AA','AB','AC','AD','AE','AF','AG','AH','AI'])])
+RowInds_BySheets = np.array([np.arange(35,36+20),
+                             np.array([23,24,26,68]),
+                             np.array([24,28,32,36,41, 25,29,33,37,42, 26,30,34,38,43, 67, 71, 
+                                       148, 149, 150, 153, 154, 155, 158, 159, 163, 164, 165, 166])])
+
+for iSheet in np.arange(len(Spreadsheets)):
+    for iColumn in ColumnInds_BySheets[iSheet]:
+        for iRow in RowInds_BySheets[iSheet]:
+            Pathways2Net0.evaluate(Spreadsheets[iSheet] + iColumn + str(iRow))
+
+# Pathways2Net0.to_file("PathwaysToNetZero_Simplified_Compiled")
+Pathways2Net0.to_file("PathwaysToNetZero_Simplified_FullOutputs_Compiled")
+
+
+start = time.time()
+# Pathways2Net0_Loaded = ExcelCompiler.from_file("PathwaysToNetZero_Simplified_Compiled")
+Pathways2Net0_Loaded = ExcelCompiler.from_file("PathwaysToNetZero_Simplified_FullOutputs_Compiled")
+end = time.time()
+print(f"INFO: took {end - start} seconds to load from serialised file")
+
+
+
+## IEV model:
+# IEVmodel = ExcelCompiler("OGTC-ORE Catapult IEV pathways economic model v01.06 FINAL - Original.xlsx")
+IEVmodel = ExcelCompiler("OGTC-ORE Catapult IEV pathways economic model v01.06 FINAL - Original Master sheet!C2=Gale.xlsx")
+
+Spreadsheets = np.array(['Master sheet!','Pathways inputs!'])
+ColumnInds_BySheets = np.array([np.array(['N','O','P','Q','R','S','T','U','V','W','X','Y', 'Z','AA','AB','AC','AD','AE','AF','AG','AH']), 
+                                np.array(['O','P','Q','R','S','T','U','V','W','X','Y','Z','AA','AB','AC','AD','AE','AF','AG','AH','AI'])])
+RowInds_BySheets = np.array([np.array([36,37,70]), 
+                             np.array([24,28,32,36,41, 25,29,33,37,42, 26,30,34,38,43, 67, 71])])
+
+for iSheet in np.arange(len(Spreadsheets)):
+    for iColumn in ColumnInds_BySheets[iSheet]:
+        for iRow in RowInds_BySheets[iSheet]:
+            IEVmodel.evaluate(Spreadsheets[iSheet] + iColumn + str(iRow))
+
+# IEVmodel.to_file("IEV_pathways_economic_model_Compiled")
+IEVmodel.to_file("IEV_pathways_economic_model_Master_sheet_C2SetToGale_Compiled")
+
+
+start = time.time()
+# IEVmodel_Loaded = ExcelCompiler.from_file("IEV_pathways_economic_model_Compiled")
+IEVmodel_Loaded = ExcelCompiler.from_file("IEV_pathways_economic_model_Master_sheet_C2SetToGale_Compiled")
+end = time.time()
+print(f"INFO: took {end - start} seconds to load from serialised file")
diff --git a/environment/fixed_policy_DirectDeployment.png b/environment/fixed_policy_DirectDeployment.png
diff --git a/environment/reference_environment_direct_deployment/.gitignore b/environment/reference_environment_direct_deployment/.gitignore
@@ -0,0 +1 @@
+.DS_Store
diff --git a/environment/reference_environment_direct_deployment/README.md b/environment/reference_environment_direct_deployment/README.md
@@ -0,0 +1,51 @@
+# A guide to the RangL environment (in OpenAI Gym)
+
+This folder contains a copy of the RangL environment which will be used for evaluation in the competition. It can be used right away to train an agent. However to gain an advantage in the competition, you can consider training your agent in a modified environment (although evaluation will still use the reference environment!). 
+
+The RangL environment is designed to be easily modified and has the following modular structure:
+
+```Dependencies```  
+
+```Helpers```  
+
+```Environment class```
+
+Any modifications should be made to the Helpers (updating the dependencies as needed). 
+
+## Helpers 
+
+The Helper classes are:
+
+* Parameters -- contains all challenge-specific parameters
+* State -- contains all state information and provides the following methods:
+    * initialise_state
+    * reset
+    * to_observation -- returns just the observations which will be passed to the agent   
+    * set_agent_prediction -- returns the predictions which will be passed to the agent
+    * is_done -- checks whether the episode has finished
+
+and the Helper functions are:
+
+* observation_space -- specifies the agent's observation space
+* action_space -- specifies the agent's action space
+* apply_action -- applies an action to the state and calculates the reward
+* verify_constraints -- checks whether the actions have violated any pre-specified constraints
+* randomise -- adds random noise to the state (representing uncertainty over the future) 
+* record -- records data for graphing at the end of the episode
+* plot_episode -- plots the recorded data
+* score -- returns score for the full episode
+
+## Environment class
+
+The environment class provides the standard initialise, reset, step and score methods required by OpenAI Gym. It also provides these additional methods:
+
+* seed -- allows the random seed to be specified
+* plot -- applies the plot_episode function 
+
+## Random seeding
+
+Explicit seeds persist upon reset() . So:
+
+*	If you set env.seed(None), reset() produces different noise
+*	If you set e.g. env.seed(42), reset() produces identical noise
+
diff --git a/environment/reference_environment_direct_deployment/__init__.py b/environment/reference_environment_direct_deployment/__init__.py
@@ -0,0 +1,5 @@
+from gym.envs.registration import register
+
+register(
+    id="reference-environment-direct-deployment-v0", entry_point="reference_environment_direct_deployment.env:GymEnv",
+)
diff --git a/environment/reference_environment_direct_deployment/__pycache__/__init__.cpython-37.pyc b/environment/reference_environment_direct_deployment/__pycache__/__init__.cpython-37.pyc
diff --git a/environment/reference_environment_direct_deployment/__pycache__/__init__.cpython-38.pyc b/environment/reference_environment_direct_deployment/__pycache__/__init__.cpython-38.pyc
diff --git a/environment/reference_environment_direct_deployment/__pycache__/env.cpython-37.pyc b/environment/reference_environment_direct_deployment/__pycache__/env.cpython-37.pyc
diff --git a/environment/reference_environment_direct_deployment/__pycache__/env.cpython-38.pyc b/environment/reference_environment_direct_deployment/__pycache__/env.cpython-38.pyc
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		PathwaysToNetZero_Simplified_Compiled.pkl
		PathwaysToNetZero_Simplified_Compiled.yml