|
| 1 | +# The Mobility House Coding Challenge |
| 2 | +This README explains all steps required to collect, process, and query information about curtailment of power generation in the balance zone of the transmission grid operator (TSO) TenneT GmbH in Germany. Data ranges from 1st of January 2020 to 30th of December 2020. |
| 3 | + |
| 4 | +- [Setting Up The Environment](#environment) |
| 5 | + - [Deploy Virtual Machine](#vm_deploy) |
| 6 | + - [Initialize Virtual Machine](#vm_init) |
| 7 | + - [Configure Database](#database_init) |
| 8 | +- [Example Code](#example_code) |
| 9 | +- [Assumptions](#assumptions) |
| 10 | + |
| 11 | +## Setting Up The Environment <a name="environment></a> |
| 12 | +Operating system: Ubuntu 22.04 LTS |
| 13 | + |
| 14 | +### Deploy Virtual Machine <a name="vm_deploy"></a> |
| 15 | +Check if you have Terraform installed by running `terraform --help` in the terminal. If Terraform is not installed on your machine follow the install guide from [Hashicorp](https://developer.hashicorp.com/terraform/install). We provide an example Terraform configuration files to deply a virtual machine running on OpenNebula. Adjust the following parts to match your environment: |
| 16 | + |
| 17 | +config.tfvars: |
| 18 | +- ON_USERNAME |
| 19 | +- ON_PASSWD |
| 20 | +- ON_GROUP |
| 21 | + |
| 22 | +on_vms.tf: |
| 23 | +- endpoint |
| 24 | +- flowpoint |
| 25 | + |
| 26 | +1. `terraform init` |
| 27 | +2. `terraform plan -var-file=config.tfvars -out=infra.out` |
| 28 | +3. `terraform apply infra.out` |
| 29 | +4. `tearraform show` to get IP address of deployed virtual machine, which is requried for the next step |
| 30 | +5. `terraform destroy -var-file=config.tfvars` (optional) |
| 31 | + |
| 32 | +### Initialize Virtual Machine <a name="vm_init"></a> |
| 33 | +Check if you have Ansible installed by running `ansible --help` in the terminal. If Ansible is not installed on your machine follow the install guide from [Ansible Documentation](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html). We install all required programs and packages on the previously deployed virtual machine via Ansible playbooks. |
| 34 | + |
| 35 | +1. Define IP address(es) by using a host group (e.g., in /etc/ansible/hosts) and refering to it in the playbooks |
| 36 | +2. Adjust host group in all following YAML files |
| 37 | +3. Update virtual machine: `ansible-playbook udpate.yml` |
| 38 | +4. Install PostgreSQL: `ansible-playbook postgres_install.yml` |
| 39 | +5. Copy files to virtual machine: `ansible-playbook copy_files.yml --extra-vars '{"path_to_files": "path/to/files/with/trailing/backslash/"}'` |
| 40 | +6. Install Python virtual environments for type= server or client: `ansible-playbook prepare_virtualenv.yml --extra-vars '{"type": "server"}'` |
| 41 | + |
| 42 | + |
| 43 | +### Configure Database <a name="database_init"></a> |
| 44 | +Set up a PostgreSQL database with two users. First, connect to PostgreSQL `sudo -u postgres psql` and list all available users with their respective priviliges `\du` / `SELECT * FROM information_schema.role_table_grants WHERE grantee='user_name'` |
| 45 | +;` or without their priviliges `SELECT usename from pg_catalog.pg_user;`. Create new users if no suitable users already exist for the server and client `CREATE USER tmh_type WITH ENCRYPTED PASSWORD 'tmh_type';`: |
| 46 | + |
| 47 | +1. type=server: User to read and write data into an existing database |
| 48 | +2. type=client: User with read-only access |
| 49 | + |
| 50 | +Preparing database and tables: |
| 51 | +1. Create database `SELECT 'CREATE DATABASE curtailment_tennet' WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'curtailment_tennet')\gexec` |
| 52 | +2. Connect to new database `\c curtailment_tennet` |
| 53 | +3. Create table: `CREATE TABLE IF NOT EXISTS curtailments (start_curtailment TIMESTAMP, end_curtailment TIMESTAMP, duration SMALLINT, level SMALLINT, cause VARCHAR, plant_id VARCHAR, operator VARCHAR, nominal_power smallint, energy numeric);` |
| 54 | +4. Change user priviliges <br> |
| 55 | +4.1 Give server and client read access `GRANT SELECT ON TABLE curtailments TO tmh_<type>;` <br> |
| 56 | +4.2 Give server write access `GRANT INSERT ON TABLE curtailments TO tmh_server;` |
| 57 | +4.3 Give server update access `GRANT UPDATE ON TABLE curtailments TO tmh_server;` |
| 58 | + |
| 59 | +## Example Code <a name="example_code"></a> |
| 60 | +``` |
| 61 | +# stdlib |
| 62 | +import os |
| 63 | +import logging |
| 64 | +
|
| 65 | +# third party |
| 66 | +import pandas as pd |
| 67 | +
|
| 68 | +# relative |
| 69 | +from tmh_server.mapping import Mapping |
| 70 | +from tmh_server.avacon_api import AvaconAPI |
| 71 | +from tmh_server.postgresql import PostgreSQL |
| 72 | +from tmh_server.process_data import ProcessData |
| 73 | +from tmh_server.mastr_scrapper import MastrScrapper |
| 74 | +
|
| 75 | +
|
| 76 | +SCRAP_MASTR: bool = False |
| 77 | +
|
| 78 | +# Initialize logger |
| 79 | +logger = logging.getLogger(__name__) |
| 80 | +logging.basicConfig(filename=os.path.join(os.getcwd(), "tmh.log"), |
| 81 | + format="%(asctime)s,%(levelname)s,%(module)s:%(funcName)s:%(lineno)s,%(message)s", |
| 82 | + datefmt="%Y-%m-%d %H:%M:%S", |
| 83 | + level=logging.DEBUG) |
| 84 | +
|
| 85 | +
|
| 86 | +def main(): |
| 87 | + """ |
| 88 | + Combines data extraction from Avacon API, cleaning the output, |
| 89 | + and storing it in a PostgreSQL database. Then scrap |
| 90 | + Marktstammdatenregister based on network operator ID to obtain |
| 91 | + mapping of EEG Anlagenschlüssel to nominal power (might be optional). |
| 92 | + Lastly, update values for nominal power and curtailed energy based |
| 93 | + on power plant ID. |
| 94 | + """ |
| 95 | + # Extract information from Avacon API based on avacon_api.json config |
| 96 | + config_path: str = os.path.join(os.path.abspath(os.path.dirname(__file__)), |
| 97 | + "configs") |
| 98 | + avacon_api: AvaconAPI = AvaconAPI(config_path=config_path, |
| 99 | + config_name="avacon_api.json") |
| 100 | + df: pd.DataFrame = avacon_api.call_api() |
| 101 | +
|
| 102 | + # Clean extracted data |
| 103 | + process_data: ProcessData = ProcessData(df) |
| 104 | + process_data.clean() |
| 105 | + df: pd.DataFrame = process_data.get_data() |
| 106 | +
|
| 107 | + # Store cleaned data in PostgreSQL database based on query.json config |
| 108 | + psql: PostgreSQL = PostgreSQL(config_path=config_path, |
| 109 | + config_name="query.json", |
| 110 | + data=df) |
| 111 | + psql.connect_and_insert() |
| 112 | +
|
| 113 | + # Scrap Marktstammdatenregister (might be optional) |
| 114 | + if SCRAP_MASTR: |
| 115 | + path_anlagenstammdaten: str = os.path.join(os.path.abspath(os.path.dirname(__file__)), |
| 116 | + "anlagenstammdaten") |
| 117 | + mapper: Mapping = Mapping(path_anlagenstammdaten, |
| 118 | + "TenneT TSO GmbH EEG-Zahlungen Bewegungsdaten 2022.csv") |
| 119 | +
|
| 120 | + scrapper: MastrScrapper = MastrScrapper() |
| 121 | + for nb_mastr_nr in mapper.get_nb_mastr_nrs(): |
| 122 | + scrapper.set_nb_mastr_nr(nb_mastr_nr) |
| 123 | + scrapper.download_via_link() |
| 124 | + scrapper.move_downloaded_file() |
| 125 | + scrapper.merge_snbs_into_one_csv(path_anlagenstammdaten) |
| 126 | +
|
| 127 | + # Update PostgreSQL database with values for nominal power and curtailed energy |
| 128 | + mapper.get_merged_snbs() |
| 129 | + mapper.create_mapping() |
| 130 | + mapper.set_df_db(psql.get_rows("curtailments")) |
| 131 | + mapper.map_power_to_plant_id() |
| 132 | + mapper.calculate_curtailed_power() |
| 133 | + mapper.calculate_curtailed_energy() |
| 134 | +
|
| 135 | + psql.close_connection() |
| 136 | + psql.set_df(mapper.df_db) |
| 137 | + psql.update_rows(col_to_update="nominal_power", |
| 138 | + col_condition="plant_id") |
| 139 | +
|
| 140 | +
|
| 141 | +if __name__ == "__main__": |
| 142 | + main() |
| 143 | +``` |
| 144 | + |
| 145 | + |
| 146 | +## Assumptions <a name="assumptions"></a> |
| 147 | +Additional information, e.g., |
| 148 | +- AVACON does not exclusively operate in TenneT area |
| 149 | +- Add code lintering |
0 commit comments