diff --git a/.github/workflows/pytest.yml b/.github/workflows/pytest.yml index 9584820..a156064 100644 --- a/.github/workflows/pytest.yml +++ b/.github/workflows/pytest.yml @@ -32,11 +32,19 @@ jobs: python-version: ${{ matrix.python-version }} - name: Install pytest - run: pip install pytest - pip install pytest-cov + run: | + pip install pytest + pip install pytest-cov - name: Install the package - run: pip install . + run: | + pip install . + + + - name: Install editable openentrance + run: | + pip uninstall --yes openentrance + pip install -e git+https://github.com/openENTRANCE/openentrance.git@main#egg=openentrance - name: Test with pytest run: pytest diff --git a/.gitignore b/.gitignore index 1fd51a2..c6ba6e1 100644 --- a/.gitignore +++ b/.gitignore @@ -58,4 +58,7 @@ MANIFEST # Outputs *.csv -!tests/fixtures/*.csv \ No newline at end of file +!tests/fixtures/*.csv + +# Dependencies +src/openentrance diff --git a/LICENSE.txt b/LICENSE.txt index 1647308..313bd2a 100755 --- a/LICENSE.txt +++ b/LICENSE.txt @@ -1,6 +1,6 @@ The MIT License (MIT) -Copyright (c) 2022 Will Usher +Copyright (c) 2022 Will Usher, Hauke Henke Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal diff --git a/README.md b/README.md index 5c792d8..c9fb73d 100644 --- a/README.md +++ b/README.md @@ -4,13 +4,21 @@ Convert OSeMOSYS results to IAMC format ## Install from Github repository - pip install git+https://github.com/osesmosys/osemosys2iamc@main#egg=osemosys2iamc +It is currently necessary to install the OpenEntrance dependency as an editable installation. +See [issue](https://github.com/openENTRANCE/openentrance/issues/202) -## Run the demo + pip install git+https://github.com/osemosys/osemosys2iamc@main#egg=osemosys2iamc + pip install -e git+https://github.com/openENTRANCE/openentrance.git@main#egg=openentrance - osemosys2iamc results config.yaml test.csv +## Run the package -Check out the IAMC formatted results in `test.csv` and plots `emissions.csv` + $ osemosys2iamc --help + osemosys2iamc + +`inputs_path`: Path to a folder of csv files (OSeMOSYS inputs). File names should correspond to OSeMOSYS parameter names. +`results_path`: Path to a folder of csv files (OSeMOSYS results). File names should correspond to OSeMOSYS variable names. +`config_path`: Path to the configuration file (see below) +`output_path`: Path to the .xlsx file you wish to write out ## The IAMC format @@ -30,7 +38,70 @@ in the contribution by Working Group III to the IPCC's Sixth Assessment Report ( Please refer to the Horizon 2020 project [openENTRANCE](https://github.com/openENTRANCE/openentrance#data-format-structure) for more information about the format and its usage in that project. -## Usage for OSeMOSYS +## Writing a configuration file + +Write a configuration file in YAML format. A simple configuration file with one result variable looks like this: + + model: OSeMBE v1.0.0 + scenario: DIAG-C400-lin-ResidualFossil + region: ['Austria'] # select countries to plot summary results + results: + - iamc_variable: 'Carbon Capture|Biomass' + tech_emi: ['(?=^.{2}(BM))^.{4}(CS)'] + emissions: [CO2] + unit: kt CO2/yr + transform: abs + osemosys_param: AnnualTechnologyEmissions + +The first section of the configuration file with the keys `model`, `scenario`, and `region` are used to define the metadata for +the IAMC template. + +The second section `results:` is where you describe each of the IAMC variables and provide instructions to osemosys2iamc on how +to compute the values. + +`iamc_variable` - this should match one of the IAMC variable names +`unit` - provide the units of the OSeMOSYS results +`transform` - only `abs` is currently available. This returns the absolute value of the results +`osemosys_param` - provide the name of the result file from which the script should extract the result data + +One or more of the following filter keys. These filter the +results by one or more columns found in OSeMOSYS results. +Following the fitering, the remaining columns except region + and year are discarded and rows are summed. + +`tech_emi` - filter the results by TECHNOLOGY and EMISSION columns using the provide regular expression and an `emissions` entry +`emissions` - a list of emissions to filter the results by the EMISSION column +`fuel` - filter by the FUEL column +`capacity` - filter the TECHNOLOGY column +`primary_technology` - filter the TECHNOLOGY column (can be replaced by `capacity` key) +`excluded_prod_tech` - filter the TECHNOLOGY column (can be replaced by `capacity` key) +`el_prod_technology` - filter the TECHNOLOGY column (can be replaced by `capacity` key) +`demand` - filters by the FUEL column (final energy) + +The value for each of these keys is a list of regular expressions. These regular expressions are used to filter the rows of data in the chosen column to those that match the +regular expression. + +Writing regular expressions can be tricky, but there are [useful tools](https://regexr.com/) to help. +Below we provide some examples: + +`^.{2}(WI)` match rows with any two characters followed by `WI` + +`(?=^.{2}(HF))^((?!00).)*$` match rows with `HF` in 3rd and 4th position which do not include `00` + +`(?=^.{2}(NG))^((?!(00)|(CS)).)*$` match rows with `NG` in 3rd and 4th position that do not include `00` or `CS` + +`^.{6}(I0)` match rows which contain any 6 characters followed by `IO` in the 7th and 8th position + +Putting this together, the following entry extracts results from the result file `ProductionByTechnologyAnnual.csv`, filters out the rows +by matching values in the TECHNOLOGY column with a list of 6 regular expressions (this is an OR operation) +and assigns the unit `PJ/yr` and adds the aggregated (summing over region and year) total under `Primary Energy` in the IAMC template format. + + - iamc_variable: 'Primary Energy' + primary_technology: ['^.{6}(I0)','^.{6}(X0)','^.{2}(HY)','^.{2}(OC)','^.{2}(SO)','^.{2}(WI)'] + unit: PJ/yr + osemosys_param: ProductionByTechnologyAnnual + +## List of relevant IAMC variables for OSeMOSYS ### Primary Energy diff --git a/requirements.txt b/requirements.txt index ac190ef..37c68a8 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,3 +1,4 @@ matplotlib pandas pyam-iamc >= 1.0 # the pyam package is released on pypi under this name +-e git+https://github.com/openENTRANCE/openentrance.git@main#egg=openentrance diff --git a/src/osemosys2iamc/resultify.py b/src/osemosys2iamc/resultify.py index 33fe399..ffc5e97 100644 --- a/src/osemosys2iamc/resultify.py +++ b/src/osemosys2iamc/resultify.py @@ -12,12 +12,15 @@ ``output_path`` is the path to the csv file written out in IAMC format """ +import functools +from multiprocessing.sharedctypes import Value +from sqlite3 import DatabaseError import pandas as pd import pyam from openentrance import iso_mapping import sys import os -from typing import List, Dict, Union +from typing import List, Dict, Optional from yaml import load, SafeLoader import matplotlib.pyplot as plt import matplotlib.dates as mdates @@ -29,90 +32,93 @@ def read_file(filename) -> pd.DataFrame: return df -def filter_var_cost(df: pd.DataFrame, technologies: List) -> pd.DataFrame: - """Return rows that match the ``technologies`` - """ - mask = df['TECHNOLOGY'].str.contains(technologies[0]) +def filter_regex(df: pd.DataFrame, patterns: List[str], column: str) -> pd.DataFrame: + """Generic filtering of rows based on columns that match a list of patterns - df['REGION'] = df['TECHNOLOGY'].str[0:2] - df = df.drop(columns=['TECHNOLOGY', 'MODE_OF_OPERATION']) + This function returns the rows where the values in a ``column`` match the + list of regular expression ``patterns`` + """ + masks = [df[column].str.match(p) for p in patterns] + return pd.concat([df[mask] for mask in masks]) + +def filter_fuels(df: pd.DataFrame, fuels: List[str]) -> pd.DataFrame: + """Returns rows which match list of regex patterns in ``technologies`` + + Parameters + ---------- + df: pd.DataFrame + The input data + fuels: List[str] + List of regex patterns + """ + return filter_regex(df, fuels, 'FUEL') - df = df[mask] +def filter_technologies(df: pd.DataFrame, technologies: List[str]) -> pd.DataFrame: + """Returns rows which match list of regex patterns in ``technologies`` - return df[df.VALUE != 0] - -def filter_fuel(df: pd.DataFrame, technologies: List, fuels: List) -> pd.DataFrame: - """Return rows which match ``technologies`` and ``fuels`` + Parameters + ---------- + df: pd.DataFrame + The input data + technologies: List[str] + List of regex patterns """ - mask = df.TECHNOLOGY.isin(technologies) - fuel_mask = df.FUEL.isin(fuels) + return filter_regex(df, technologies, 'TECHNOLOGY') - return df[mask & fuel_mask] - -def filter_emission(df: pd.DataFrame, emission: List) -> pd.DataFrame: - """Return rows which match ``emission`` and fill region name from technology +def filter_technology_fuel(df: pd.DataFrame, technologies: List, fuels: List) -> pd.DataFrame: + """Return rows which match ``technologies`` and ``fuels`` """ + df = filter_technologies(df, technologies) + df = filter_fuels(df, fuels) - mask = df.EMISSION.isin(emission) + df = df.groupby(by=['REGION','YEAR'], as_index=False)["VALUE"].sum() + return df[df.VALUE != 0] - # First two characters in technology match ISO2 country name - df['REGION'] = df['TECHNOLOGY'].str[0:2] - df = df.drop(columns='TECHNOLOGY') +def filter_emission_tech(df: pd.DataFrame, emission: List[str], technologies: Optional[List[str]]=None) -> pd.DataFrame: + """Return annual emissions or captured emissions by one or several technologies. - return df[mask] + Parameters + ---------- + df: pd.DataFrame + emission: List[str] + List of regex patterns + technologies: List[str], default=None + List of regex patterns -def filter_emission_tech(df: pd.DataFrame, tech: List, emission: List) -> pd.DataFrame: - """Return a dataframe with annual emissions or captured emissions by one or several technologies. + Returns + ------- + pandas.DataFrame """ - mask_emi = df.EMISSION.isin(emission) - df = df[mask_emi] + df['REGION'] = df['TECHNOLOGY'].str[:2] + df = filter_regex(df, emission, 'EMISSION') - df_f = pd.DataFrame(columns=['REGION','TECHNOLOGY','EMISSION','YEAR','VALUE']) + if technologies: + # Create a list of masks, one for each row that matches the pattern listed in ``tech`` + df = filter_technologies(df, technologies) - for t in range(len(tech)): - mask_tech = df['TECHNOLOGY'].str.contains(tech[t]) - df_t = df[mask_tech] - df_f = df_f.append(df_t) + df = df.groupby(by=['REGION','YEAR'], as_index=False)["VALUE"].sum() + return df[df.VALUE != 0] - df_f['REGION'] = df_f['TECHNOLOGY'].str[:2] - df = df_f.drop(columns='TECHNOLOGY') +def filter_capacity(df: pd.DataFrame, technologies: List[str]) -> pd.DataFrame: + """Return aggregated rows filtered on technology column. - df['VALUE'] = df['VALUE']*(-1) + Parameters + ---------- + df: pd.DataFrame + The input data + technologies: List[str] + List of regex patterns - return df - -def filter_capacity(df: pd.DataFrame, technologies: List) -> pd.DataFrame: - """Return rows that indicate the installed power generation capacity. - """ - df['REGION'] = df['TECHNOLOGY'].str[:2] - df_f = pd.DataFrame(columns=['REGION','TECHNOLOGY','YEAR','VALUE']) - for t in range(len(technologies)): - mask = df['TECHNOLOGY'].str.contains(technologies[t]) - df_t = df[mask] - df_f = df_f.append(df_t) - - df = pd.DataFrame(columns=["REGION","YEAR","VALUE"]) - for r in df_f["REGION"].unique(): - for y in df_f["YEAR"].unique(): - df = df.append({"REGION": r, "YEAR": y, "VALUE": df_f.loc[(df_f["REGION"]==r)&(df_f["YEAR"]==y),["VALUE"]].sum(axis=0).VALUE},ignore_index=True).sort_values(by=['REGION','YEAR']) - return df[df.VALUE != 0].reset_index(drop=True) - -def filter_ProdByTechAn(df: pd.DataFrame, technologies: List) -> pd.DataFrame: - """Return rows that indicate Primary Energy use/generation + Returns + ------- + pandas.DataFrame """ df['REGION'] = df['TECHNOLOGY'].str[:2] - df_f = pd.DataFrame(columns=['REGION','TECHNOLOGY','FUEL','YEAR','VALUE']) - for t in range(len(technologies)): - mask = df['TECHNOLOGY'].str.contains(technologies[t]) - df_t = df[mask] - df_f = df_f.append(df_t) - - df = pd.DataFrame(columns=["REGION","YEAR","VALUE"]) - for r in df_f["REGION"].unique(): - for y in df_f["YEAR"].unique(): - df = df.append({"REGION": r, "YEAR": y, "VALUE": df_f.loc[(df_f["REGION"]==r)&(df_f["YEAR"]==y),["VALUE"]].sum(axis=0).VALUE},ignore_index=True).sort_values(by=['REGION','YEAR']) - return df[df.VALUE != 0].reset_index(drop=True) + df = filter_technologies(df, technologies) + + df = df.groupby(by=['REGION','YEAR'], as_index=False)["VALUE"].sum() + return df[df.VALUE != 0] def filter_final_energy(df: pd.DataFrame, fuels: List) -> pd.DataFrame: """Return dataframe that indicate the final energy demand/use per country and year. @@ -123,65 +129,22 @@ def filter_final_energy(df: pd.DataFrame, fuels: List) -> pd.DataFrame: exit(1) df['REGION'] = df['FUEL'].str[:2] - df['FUEL'] = df['FUEL'].str[2:] - df_f = pd.DataFrame(columns=['REGION','TIMESLICE','FUEL','YEAR','VALUE']) - for f in range(len(fuels)): - mask = df['FUEL'].str.contains(fuels[f]) - df_t = df[mask] - df_f = df_f.append(df_t) + df['FUEL'] = df['FUEL'].str[2:] + df_f = filter_fuels(df, fuels) - df = pd.DataFrame(columns=['REGION','YEAR','VALUE']) - for r in df_f['REGION'].unique(): - for y in df_f['YEAR'].unique(): - df = df.append({"REGION": r, "YEAR": y, "VALUE": df_f.loc[(df_f["REGION"]==r)&(df_f["YEAR"]==y),["VALUE"]].sum(axis=0).VALUE},ignore_index=True).sort_values(by=['REGION','YEAR']) - return df[df.VALUE != 0].reset_index(drop=True) + df = df_f.groupby(by=['REGION','YEAR'], as_index=False)["VALUE"].sum() + return df[df.VALUE != 0] def calculate_trade(results: dict, techs: List) -> pd.DataFrame: """Return dataframe with the net exports of a commodity """ - countries = pd.Series(dtype='object') - years = pd.Series() - for p in results: - df = results[p] - df_f = pd.DataFrame(columns=df.columns) - for t in techs: - mask = df['TECHNOLOGY'].str.contains(t) - df_t = df[mask] - df_f = df_f.append(df_t) - - df_f['REGION'] = df_f['FUEL'].str[:2] - df_f = df_f.drop(columns='FUEL') - df_f = df_f.groupby(by=['REGION', 'YEAR']).sum() - df_f = df_f.reset_index(level=['REGION', 'YEAR']) - - countries = countries.append(df_f.loc[:,'REGION']) - years = years.append(df_f.loc[:,'YEAR']) - results[p] = df_f - - countries = countries.unique() - years = years.unique() - exports = results['UseByTechnology'] - imports = results['ProductionByTechnologyAnnual'] - - df = pd.DataFrame(columns=['REGION','YEAR','VALUE']) - for country in countries: - for year in years: - if not exports[(exports['REGION']==country)&(exports['YEAR']==year)].empty: - if not imports[(imports['REGION']==country)&(imports['YEAR']==year)].empty: - value = exports[(exports['REGION']==country)&(exports['YEAR']==year)].iloc[0]['VALUE']-imports[(imports['REGION']==country)&(imports['YEAR']==year)].iloc[0]['VALUE'] - df = df.append({'REGION': country, 'YEAR': year, 'VALUE': value}, ignore_index=True) - else: - value = exports[(exports['REGION']==country)&(exports['YEAR']==year)].iloc[0]['VALUE'] - df = df.append({'REGION': country, 'YEAR': year, 'VALUE': value}, ignore_index=True) - else: - if not imports[(imports['REGION']==country)&(imports['YEAR']==year)].empty: - value = -imports[(imports['REGION']==country)&(imports['YEAR']==year)].iloc[0]['VALUE'] - df = df.append({'REGION': country, 'YEAR': year, 'VALUE': value}, ignore_index=True) - + exports = filter_capacity(results['UseByTechnology'], techs).set_index(['REGION', 'YEAR']) + imports = filter_capacity(results['ProductionByTechnologyAnnual'], techs).set_index(['REGION', 'YEAR']) + df = exports.subtract(imports, fill_value=0) - return df + return df.reset_index() def extract_results(df: pd.DataFrame, technologies: List) -> pd.DataFrame: """Return rows which match ``technologies`` @@ -191,41 +154,6 @@ def extract_results(df: pd.DataFrame, technologies: List) -> pd.DataFrame: return df[mask] -def aggregate(data: pd.DataFrame): - """Sums rows while grouping regions and years - """ - - return data.groupby(by=['REGION', 'YEAR']).sum() - -def make_iamc(data: pd.DataFrame, - iam_model: str, - iam_scenario: str, - iam_variable: str, - iam_unit: str - ) -> pyam.IamDataFrame: - """Creates an IAM Dataframe from raw data - - Arguments - --------- - data: pd.DataFrame - Contains columns for region, year and value - iam_model: str - The model name to insert into the IAMC dataframe - iam_scenario: str - The scenario name to insert into the IAMC dataframe - iam_variable: str - The IAMC variable name to insert into the IAMC dataframe - iam_unit: str - The unit to insert into the IAMC dataframe - - """ - return pyam.IamDataFrame( - data, - model=iam_model, - scenario=iam_scenario, - variable=iam_variable, - unit=iam_unit, - ) def load_config(filepath: str) -> Dict: """Reads the configuration file @@ -239,7 +167,7 @@ def load_config(filepath: str) -> Dict: config = load(configfile, Loader=SafeLoader) return config -def make_plots(df, model: str, scenario: str, regions: list): +def make_plots(df: pyam.IamDataFrame, model: str, scenario: str, regions: List[str]): """Creates standard plots Arguments @@ -254,26 +182,28 @@ def make_plots(df, model: str, scenario: str, regions: list): # regions = config['region']#for testing args = dict(model=model, scenario=scenario) - print(args) + print(args, regions) - for region in regions: + fig, ax = plt.subplots() + + assert isinstance(regions, list) - # Plot primary energy - fig, ax = plt.subplots() - print(ax) - pe = df.filter(**args, variable='Primary Energy|*', region=region) - if pe: + for region in regions: + assert isinstance(region, str) + print(f"Plotting {region}") + # Plot primary energy + data = df.filter(**args, variable='Primary Energy|*', region=region) # type: pyam.IamDataFrame + if data: + print(data) locator = mdates.AutoDateLocator(minticks=10) #locator.intervald['YEARLY'] = [10] - pe.plot.bar(ax=ax, stacked=True, title='Primary energy mix %s' % region) + data.plot.bar(ax=ax, stacked=True, title='Primary energy mix %s' % region) plt.legend(bbox_to_anchor=(0.,-0.5), loc='upper left') plt.tight_layout() ax.xaxis.set_major_locator(locator) fig.savefig('primary_energy_%s.pdf' % region, bbox_inches='tight', transparent=True, pad_inches=0) - - # Plot secondary energy (electricity generation) - fig, ax = plt.subplots() - print(ax) + plt.clf() + # Plot secondary energy (electricity generation) se = df.filter(**args, variable='Secondary Energy|Electricity|*', region=region) if se: locator = mdates.AutoDateLocator(minticks=10) @@ -283,27 +213,26 @@ def make_plots(df, model: str, scenario: str, regions: list): plt.tight_layout() ax.xaxis.set_major_locator(locator) fig.savefig('electricity_generation_%s.pdf' % region, bbox_inches='tight', transparent=True, pad_inches=0) - - # Create generation capacity plot - fig, ax = plt.subplots() + plt.clf() + # Create generation capacity plot cap = df.filter(**args, variable='Capacity|Electricity|*', region=region) if cap: cap.plot.bar(ax=ax, stacked=True, title='Generation Capacity %s' % region) plt.legend(bbox_to_anchor=(0.,-0.25),loc='upper left') plt.tight_layout() fig.savefig('capacity_%s.pdf' % region, bbox_inches='tight', transparent=True, pad_inches=0) + plt.clf() # Create emissions plot emi = df.filter(**args, variable="Emissions|CO2*").filter(region="World", keep=False) - print(emi) + # print(emi) if emi: - fig, ax = plt.subplots() emi.plot.bar(ax=ax, bars="region", stacked=True, title="CO2 emissions by region", cmap="tab20" ) plt.legend(bbox_to_anchor=(1.,1.05),loc='upper left', ncol=2) fig.savefig('emission.pdf', bbox_inches='tight', transparent=True, pad_inches=0) - + plt.clf() def main(config: Dict, inputs_path: str, results_path: str) -> pyam.IamDataFrame: @@ -316,97 +245,153 @@ def main(config: Dict, inputs_path: str, results_path: str) -> pyam.IamDataFrame --------- config : dict The configuration dictionary + inputs_path: str + Path to a folder of csv files (OSeMOSYS inputs) + results_path: str + Path to a folder of csv files (OSeMOSYS results) """ blob = [] - for input in config['inputs']: - - inpathname = os.path.join(inputs_path, input['osemosys_param'] + '.csv') - inputs = read_file(inpathname) - - unit = input['unit'] - - technologies = input['variable_cost'] - data = filter_var_cost(inputs, technologies) - - aggregated = aggregate(data) - - if not aggregated.empty: - iamc = make_iamc(aggregated, config['model'], config['scenario'], input['iamc_variable'], unit) - blob.append(iamc) - - for result in config['results']: - - if type(result['osemosys_param']) == str: - inpathname = os.path.join(results_path, result['osemosys_param'] + '.csv') - results = read_file(inpathname) - - try: - technologies = result['technology'] - except KeyError: - pass - unit = result['unit'] - if 'fuel' in result.keys(): - fuels = result['fuel'] - data = filter_fuel(results, technologies, fuels) - elif 'emission' in result.keys(): - emission = result['emission'] - data = filter_emission(results, emission) - elif 'tech_emi' in result.keys(): - emission = result['emissions'] - technologies = result['tech_emi'] - data = filter_emission_tech(results, technologies, emission) - elif 'capacity' in result.keys(): - technologies = result['capacity'] - data = filter_capacity(results, technologies) - elif 'primary_technology' in result.keys(): - technologies = result['primary_technology'] - data = filter_ProdByTechAn(results, technologies) - elif 'excluded_prod_tech' in result.keys(): - technologies = result['excluded_prod_tech'] - data = filter_ProdByTechAn(results, technologies) - elif 'el_prod_technology' in result.keys(): - technologies = result['el_prod_technology'] - data = filter_ProdByTechAn(results, technologies) - elif 'demand' in result.keys(): - demands = result['demand'] - data = filter_final_energy(results, demands) - else: - data = extract_results(results, technologies) + try: + for input in config['inputs']: + + inpathname = os.path.join(inputs_path, input['osemosys_param'] + '.csv') + inputs = read_file(inpathname) + + unit = input['unit'] + + technologies = input['variable_cost'] + data = filter_capacity(inputs, technologies) + + if not data.empty: + data = data.rename(columns={'REGION': 'region', + 'YEAR': 'year', + 'VALUE': 'value'}) + iamc = pyam.IamDataFrame( + data, + model=config['model'], + scenario=config['scenario'], + variable=input['iamc_variable'], + unit=unit) + blob.append(iamc) + except KeyError: + pass + + try: + for result in config['results']: + + if type(result['osemosys_param']) == str: + path_name = os.path.join(results_path, result['osemosys_param'] + '.csv') + results = read_file(path_name) + + try: + technologies = result['technology'] + except KeyError: + pass + unit = result['unit'] + if 'fuel' in result.keys(): + fuels = result['fuel'] + data = filter_technology_fuel(results, technologies, fuels) + elif 'emissions' in result.keys(): + if 'tech_emi' in result.keys(): + emission = result['emissions'] + technologies = result['tech_emi'] + data = filter_emission_tech(results, emission, technologies) + else: + emission = result['emissions'] + data = filter_emission_tech(results, emission) + elif 'capacity' in result.keys(): + technologies = result['capacity'] + data = filter_capacity(results, technologies) + elif 'primary_technology' in result.keys(): + technologies = result['primary_technology'] + data = filter_capacity(results, technologies) + elif 'excluded_prod_tech' in result.keys(): + technologies = result['excluded_prod_tech'] + data = filter_capacity(results, technologies) + elif 'el_prod_technology' in result.keys(): + technologies = result['el_prod_technology'] + data = filter_capacity(results, technologies) + elif 'demand' in result.keys(): + demands = result['demand'] + data = filter_final_energy(results, demands) + else: + data = extract_results(results, technologies) - else: - results = {} - for p in result['osemosys_param']: - inpathname = os.path.join(results_path, p + '.csv') - results[p] = read_file(inpathname) - if 'trade_tech' in result.keys(): - technologies = result['trade_tech'] - data = calculate_trade(results, technologies) + else: + results = {} + for p in result['osemosys_param']: + path_name = os.path.join(results_path, p + '.csv') + results[p] = read_file(path_name) + if 'trade_tech' in result.keys(): + technologies = result['trade_tech'] + data = calculate_trade(results, technologies) + + if 'transform' in result.keys(): + if result['transform'] == 'abs': + data['VALUE'] = data['VALUE'].abs() + else: + pass + + if not data.empty: + data = data.rename(columns={'REGION': 'region', + 'YEAR': 'year', + 'VALUE': 'value'}) + # print(data) + iamc = pyam.IamDataFrame( + data, + model=config['model'], + scenario=config['scenario'], + variable=result['iamc_variable'], + unit=unit) + blob.append(iamc) + except KeyError: + pass - aggregated = aggregate(data) + all_data = pyam.concat(blob) - if not aggregated.empty: - iamc = make_iamc(aggregated, config['model'], config['scenario'], result['iamc_variable'], unit) - blob.append(iamc) + all_data = all_data.convert_unit('PJ/yr', to='EJ/yr') + all_data = all_data.convert_unit('ktCO2/yr', to='Mt CO2/yr', factor=0.001) + all_data = all_data.convert_unit('MEUR_2015/PJ', to='EUR_2020/GJ', factor=1.05) + all_data = all_data.convert_unit('kt CO2/yr', to='Mt CO2/yr') - all_data = pyam.concat(blob) + if iso_mapping is not None: + data = all_data.timeseries() + data.index = data.index.set_levels(data.index.levels[2].map(iso_mapping), level=2) + all_data = pyam.IamDataFrame(data) + else: + msg = "Please ensure that the openentrance package is installed as an editable library " \ + "See https://github.com/openENTRANCE/openentrance/issues/202" + raise ValueError(msg) - all_data = all_data.convert_unit('PJ/yr', to='EJ/yr').timeseries() - all_data = pyam.IamDataFrame(all_data) - all_data = all_data.convert_unit('ktCO2/yr', to='Mt CO2/yr', factor=0.001).timeseries() - all_data = pyam.IamDataFrame(all_data) - all_data = all_data.convert_unit('MEUR_2015/PJ', to='EUR_2020/GJ', factor=1.05).timeseries() - all_data = pyam.IamDataFrame(all_data) - all_data = all_data.convert_unit('kt CO2/yr', to='Mt CO2/yr').timeseries() - all_data.index = all_data.index.set_levels(all_data.index.levels[2].map(iso_mapping), level=2) all_data = pyam.IamDataFrame(all_data) return all_data + +def aggregate(func): + """Decorator for filters which returns the aggregated data + + """ + @functools.wraps(func) + def wrapper(*args, **kwargs): + # Get the dataframe from the filter + data = func(*args, **kwargs) + # Apply the aggregation + data = data.groupby(by=['REGION', 'YEAR']).sum() + # Make the IAMDataFrame + return pyam.IamDataFrame( + data, + model=iam_model, + scenario=iam_scenario, + variable=iam_variable, + unit=iam_unit) + return wrapper + def entry_point(): args = sys.argv[1:] if len(args) != 4: - print("Usage: python resultify.py ") + print("Usage: osemosys2iamc ") exit(1) inputs_path = args[0] @@ -421,11 +406,13 @@ def entry_point(): model = config['model'] scenario = config['scenario'] regions = config['region'] - make_plots(all_data, model, scenario, regions) + + # Plotting fail reported in [issue 25](https://github.com/OSeMOSYS/osemosys2iamc/issues/25) + # make_plots(all_data, model, scenario, regions) all_data.to_excel(outpath, sheet_name='data') if __name__ == "__main__": - entry_point() \ No newline at end of file + entry_point() diff --git a/tests/fixtures/VariableCost.csv b/tests/fixtures/VariableCost.csv index 0dd256f..c045b24 100644 --- a/tests/fixtures/VariableCost.csv +++ b/tests/fixtures/VariableCost.csv @@ -1,15 +1,15 @@ REGION,TECHNOLOGY,MODE_OF_OPERATION,YEAR,VALUE -REGION1,ATBM00I00,1,2015,4.655510125 -REGION1,ATBM00I00,1,2016,4.655510125 -REGION1,ATBM00X00,1,2015,2.327755063 -REGION1,ATBM00X00,1,2016,2.327755063 -REGION1,ATBMCCPH1,1,2015,0.008364 -REGION1,ATBMCCPH1,1,2016,0.008364 -REGION1,ATOI00I00,1,2015,8.848033205 -REGION1,ATOI00I00,1,2016,9.428685384 -REGION1,ATOI00X00,1,2015,7.963229885 -REGION1,ATOI00X00,1,2016,8.485816846 -REGION1,BEBM00I00,1,2015,4.655510125 -REGION1,BEBM00I00,1,2016,4.655510125 -REGION1,BEBM00X00,1,2015,2.327755063 -REGION1,BEBM00X00,1,2016,2.327755063 \ No newline at end of file +REGION1,ATBM00I00,1,2015,1.0 +REGION1,ATBM00I00,1,2016,2.0 +REGION1,ATBM00X00,1,2015,3.0 +REGION1,ATBM00X00,1,2016,4.0 +REGION1,ATBMCCPH1,1,2015,5.0 +REGION1,ATBMCCPH1,1,2016,6.0 +REGION1,ATOI00I00,1,2015,1.1 +REGION1,ATOI00I00,1,2016,1.2 +REGION1,ATOI00X00,1,2015,1.3 +REGION1,ATOI00X00,1,2016,1.4 +REGION1,BEBM00I00,1,2015,1.5 +REGION1,BEBM00I00,1,2016,1.6 +REGION1,BEBM00X00,1,2015,1.7 +REGION1,BEBM00X00,1,2016,1.8 \ No newline at end of file diff --git a/tests/fixtures/config_input.yaml b/tests/fixtures/config_input.yaml new file mode 100644 index 0000000..aa8091d --- /dev/null +++ b/tests/fixtures/config_input.yaml @@ -0,0 +1,8 @@ +model: OSeMBE v1.0.0 +scenario: DIAG-C400-lin-ResidualFossil +region: ['Austria'] #, 'Belgium', 'Bulgaria', 'Switzerland', 'Cyprus', 'Czech Republic', 'Germany', 'Denmark', 'Estonia', 'Spain', 'Finland', 'France', 'Greece', 'Croatia', 'Hungary', 'Ireland', 'Italy', 'Lithuania', 'Luxembourg', 'Latvia', 'Malta', 'The Netherlands', 'Norway', 'Poland', 'Portugal', 'Romania', 'Sweden', 'Slovenia', 'Slovakia', 'United Kingdom',] +inputs: +- iamc_variable: 'Price|Primary Energy|Biomass' + variable_cost: ['(?=^.{2}(BM))^.{6}(X0)'] + unit: MEUR_2015/PJ + osemosys_param: 'VariableCost' diff --git a/tests/fixtures/config_result.yaml b/tests/fixtures/config_result.yaml new file mode 100644 index 0000000..f8bf04a --- /dev/null +++ b/tests/fixtures/config_result.yaml @@ -0,0 +1,8 @@ +model: OSeMBE v1.0.0 +scenario: DIAG-C400-lin-ResidualFossil +region: ['Austria'] #, 'Belgium', 'Bulgaria', 'Switzerland', 'Cyprus', 'Czech Republic', 'Germany', 'Denmark', 'Estonia', 'Spain', 'Finland', 'France', 'Greece', 'Croatia', 'Hungary', 'Ireland', 'Italy', 'Lithuania', 'Luxembourg', 'Latvia', 'Malta', 'The Netherlands', 'Norway', 'Poland', 'Portugal', 'Romania', 'Sweden', 'Slovenia', 'Slovakia', 'United Kingdom',] +results: +- iamc_variable: 'Capacity|Electricity' + capacity: ['^((?!(EL)|(00)).)*$'] + unit: GW + osemosys_param: TotalCapacityAnnual diff --git a/tests/fixtures/config_result_capture.yaml b/tests/fixtures/config_result_capture.yaml new file mode 100644 index 0000000..2e66b38 --- /dev/null +++ b/tests/fixtures/config_result_capture.yaml @@ -0,0 +1,10 @@ +model: OSeMBE v1.0.0 +scenario: DIAG-C400-lin-ResidualFossil +region: ['Austria'] #, 'Belgium', 'Bulgaria', 'Switzerland', 'Cyprus', 'Czech Republic', 'Germany', 'Denmark', 'Estonia', 'Spain', 'Finland', 'France', 'Greece', 'Croatia', 'Hungary', 'Ireland', 'Italy', 'Lithuania', 'Luxembourg', 'Latvia', 'Malta', 'The Netherlands', 'Norway', 'Poland', 'Portugal', 'Romania', 'Sweden', 'Slovenia', 'Slovakia', 'United Kingdom',] +results: +- iamc_variable: 'Carbon Capture|Biomass' + tech_emi: ['(?=^.{2}(BM))^.{4}(CS)'] + emissions: [CO2] + unit: kt CO2/yr + transform: abs + osemosys_param: AnnualTechnologyEmissions diff --git a/tests/test_cli.py b/tests/test_cli.py new file mode 100644 index 0000000..ea821ef --- /dev/null +++ b/tests/test_cli.py @@ -0,0 +1,31 @@ +from pytest import mark + +import os +from subprocess import run +from tempfile import NamedTemporaryFile, mkdtemp + +class TestCLI: + + def test_convert_commands(self): + + temp = mkdtemp() + config_path = os.path.join("tests", "fixtures", "config_result.yaml") + input_path = os.path.join("tests", "fixtures") + results_path = os.path.join("tests", "fixtures") + + iamc_target = os.path.join("test_iamc.xlsx") + + commands = [ + "osemosys2iamc", + input_path, + results_path, + config_path, + iamc_target + ] + + expected = "" + + actual = run(commands, capture_output=True) + assert expected in str(actual.stdout) + print(" ".join(commands)) + assert actual.returncode == 0, print(actual.stdout) diff --git a/tests/test_main.py b/tests/test_main.py new file mode 100644 index 0000000..9489e7f --- /dev/null +++ b/tests/test_main.py @@ -0,0 +1,86 @@ +from osemosys2iamc.resultify import main +import os +from yaml import load, SafeLoader +from pyam import IamDataFrame +from pyam.testing import assert_iamframe_equal +import pandas as pd + +def test_main_input(): + + config = os.path.join("tests", "fixtures", "config_input.yaml") + inputs = os.path.join("tests", "fixtures") + results = os.path.join("tests", "fixtures") + + with open(config, 'r') as config_file: + config = load(config_file, Loader=SafeLoader) + + actual = main(config, inputs, results) + + data = pd.DataFrame([ + ['Austria', 'Price|Primary Energy|Biomass', 2015, 3.15], + ['Austria', 'Price|Primary Energy|Biomass', 2016, 4.2], + ['Belgium', 'Price|Primary Energy|Biomass', 2015, 1.785], + ['Belgium', 'Price|Primary Energy|Biomass', 2016, 1.890], + ], columns=['region', 'variable', 'year', 'value']) + + expected = IamDataFrame(data, model='OSeMBE v1.0.0', scenario= 'DIAG-C400-lin-ResidualFossil', unit='EUR_2020/GJ') + + assert_iamframe_equal(actual, expected) + +def test_main_result(): + + config = os.path.join("tests", "fixtures", "config_result.yaml") + inputs = os.path.join("tests", "fixtures") + results = os.path.join("tests", "fixtures") + + with open(config, 'r') as config_file: + config = load(config_file, Loader=SafeLoader) + + actual = main(config, inputs, results) + + data = pd.DataFrame([ + ['Austria','Capacity|Electricity',2015,0.446776], + ['Belgium','Capacity|Electricity',2016,0.184866], + ['Bulgaria','Capacity|Electricity',2015,4.141], + ['Switzerland','Capacity|Electricity',2026,0.004563975391582646], + ['Cyprus','Capacity|Electricity',2015,0.3904880555817921], + ['Czech Republic','Capacity|Electricity',2015,0.299709], + ['Germany','Capacity|Electricity',2015,9.62143], + ['Denmark','Capacity|Electricity',2015,0.0005], + ['Estonia','Capacity|Electricity',2015,0.006], + ['Spain','Capacity|Electricity',2015,7.7308], + ['Finland','Capacity|Electricity',2015,0.0263], + ['France','Capacity|Electricity',2015,0.47835], + ], columns=['region', 'variable', 'year', 'value']) + + expected = IamDataFrame(data, model='OSeMBE v1.0.0', scenario= 'DIAG-C400-lin-ResidualFossil', unit='GW') + + assert_iamframe_equal(actual, expected) + +def test_main_result_capture(): + """ + REGION1,ATBMCSPN2,CO2,2026,-7573.069442598169 + REGION1,ATBMCSPN2,CO2,2027,-7766.777427515737 + REGION1,BEBMCSPN2,CO2,2026,-2244.98280006968 + REGION1,BEBMCSPN2,CO2,2027,-6746.886436926597 + """ + + config = os.path.join("tests", "fixtures", "config_result_capture.yaml") + inputs = os.path.join("tests", "fixtures") + results = os.path.join("tests", "fixtures") + + with open(config, 'r') as config_file: + config = load(config_file, Loader=SafeLoader) + + actual = main(config, inputs, results) + + data = pd.DataFrame([ + ['Austria', 'Carbon Capture|Biomass', 2026, 7.573069442598169], + ['Austria', 'Carbon Capture|Biomass', 2027, 7.766777427515737], + ['Belgium', 'Carbon Capture|Biomass', 2026, 2.24498280006968], + ['Belgium', 'Carbon Capture|Biomass', 2027, 6.746886436926597], + ], columns=['region', 'variable', 'year', 'value']) + + expected = IamDataFrame(data, model='OSeMBE v1.0.0', scenario= 'DIAG-C400-lin-ResidualFossil', unit='Mt CO2/yr') + + assert_iamframe_equal(actual, expected) \ No newline at end of file diff --git a/tests/test_resultify.py b/tests/test_resultify.py index a39dea0..34be3e9 100644 --- a/tests/test_resultify.py +++ b/tests/test_resultify.py @@ -2,7 +2,49 @@ import pandas as pd import os import pytest -from osemosys2iamc.resultify import extract_results, filter_var_cost, filter_fuel, filter_emission, filter_emission_tech, filter_ProdByTechAn, filter_final_energy, filter_capacity +from osemosys2iamc.resultify import (filter_technology_fuel, + filter_emission_tech, + filter_final_energy, + filter_capacity, + calculate_trade) + + +class TestTrade: + + def test_trade(self): + + use = [ + ['REGION1','ID','ATBM00X00','ATBM', 2014, 5.0], + ['REGION1','ID','ATBM00X00','ATBM', 2015, 5.0], + ] + + production = [ + ['REGION1','ATBM00X00','ATBM', 2015, 10.0], + ['REGION1','ATBM00X00','ATBM', 2016, 10.0], + ] + + results = { + 'UseByTechnology': pd.DataFrame( + data = use, + columns = ['REGION','TIMESLICE','TECHNOLOGY','FUEL','YEAR','VALUE'] + ), + 'ProductionByTechnologyAnnual': pd.DataFrame( + data = production, + columns=['REGION','TECHNOLOGY','FUEL','YEAR','VALUE']) + } + + techs = ['ATBM00X00'] + + actual = calculate_trade(results, techs) + + expected_data = [ + ['AT',2014, 5.0], + ['AT',2015, -5.0], + ['AT',2016, -10.0], + ] + + expected = pd.DataFrame(expected_data, columns=['REGION', 'YEAR', 'VALUE']) + pd.testing.assert_frame_equal(actual, expected) class TestEmissions: @@ -13,28 +55,26 @@ def test_filter_emission(self): input_data = pd.read_csv(filepath) emission = ['CO2'] - actual = filter_emission(input_data, emission) + actual = filter_emission_tech(input_data, emission) data = [ - ['AT', 'CO2', 2026, -7573.069442598169], - ['AT', 'CO2', 2027, -7766.777427515737], - ['AT', 'CO2', 2030, 3043.14883455963], - ['AT', 'CO2', 2031, 2189.064680841067], - ['AT', 'CO2', 2032, 2315.8212665203155], - ['AT', 'CO2', 2026, 1328.206881170592], - ['AT', 'CO2', 2027, 1237.245344603424], - ['BE', 'CO2', 2026, -2244.98280006968], - ['BE', 'CO2', 2027, -6746.886436926597], - ["BG", 'CO2', 2030, 11096.55693088164], - ["BG", 'CO2', 2031, 11069.257140908643], - ["BG", 'CO2', 2032, 11041.957354265856], + ["AT", 2026, -6244.862561], + ["AT", 2027, -6529.532083], + ["AT", 2030, 3043.148835], + ["AT", 2031, 2189.064681], + ["AT", 2032, 2315.821267], + ["BE", 2026, -2244.982800], + ["BE", 2027, -6746.886437], + ["BG", 2030, 11096.556931], + ["BG", 2031, 11069.257141], + ["BG", 2032, 11041.957354] ] - expected = pd.DataFrame(data=data, columns=['REGION', 'EMISSION', 'YEAR', 'VALUE']) - - index = ['REGION', 'EMISSION', 'YEAR'] + expected = pd.DataFrame( + data=data, + columns=['REGION', 'YEAR', 'VALUE']) - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) + pd.testing.assert_frame_equal(actual, expected) def test_filter_tech_emission(self): @@ -43,20 +83,21 @@ def test_filter_tech_emission(self): tech = ['(?=^.{2}(BM))^.{4}(CS)'] emission = ['CO2'] - actual = filter_emission_tech(input_data, tech, emission) + actual = filter_emission_tech(input_data, emission, tech) data = [ - ['AT', 'CO2', 2026, 7573.069442598169], - ['AT', 'CO2', 2027, 7766.777427515737], - ['BE', 'CO2', 2026, 2244.98280006968], - ['BE', 'CO2', 2027, 6746.886436926597], + ['AT', 2026, -7573.069442598169], + ['AT', 2027, -7766.777427515737], + ['BE', 2026, -2244.98280006968], + ['BE', 2027, -6746.886436926597], ] - expected = pd.DataFrame(data=data, columns=['REGION', 'EMISSION', 'YEAR', 'VALUE']) - - index = ['REGION', 'EMISSION', 'YEAR'] - - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) + expected = pd.DataFrame( + data=data, + columns=['REGION', 'YEAR', 'VALUE']) + print(actual) + print(expected) + pd.testing.assert_frame_equal(actual, expected) class TestFilter: @@ -66,34 +107,29 @@ def test_filter_fuel(self): technologies = ['ALUPLANT'] fuels = ['C1_P_HCO'] - actual = filter_fuel(input_data, technologies, fuels) + actual = filter_technology_fuel(input_data, technologies, fuels) data = [ - ['Globe', 'ID', 'ALUPLANT', 'C1_P_HCO', 2010, 0.399179303], - ['Globe', 'ID', 'ALUPLANT', 'C1_P_HCO', 2011, 0.397804018], - ['Globe', 'ID', 'ALUPLANT', 'C1_P_HCO', 2012, 0.390495285], - ['Globe', 'IN', 'ALUPLANT', 'C1_P_HCO', 2010, 0.399179303], - ['Globe', 'IN', 'ALUPLANT', 'C1_P_HCO', 2011, 0.397804018], - ['Globe', 'IN', 'ALUPLANT', 'C1_P_HCO', 2012, 0.390495285], - ['Globe', 'IP', 'ALUPLANT', 'C1_P_HCO', 2010, 0.029739228], - ['Globe', 'IP', 'ALUPLANT', 'C1_P_HCO', 2011, 0.029739228], - ['Globe', 'IP', 'ALUPLANT', 'C1_P_HCO', 2012, 0.07106111], + ["Globe", 2010, 0.828098], + ["Globe", 2011, 0.825347], + ["Globe", 2012, 0.852052] ] + expected = pd.DataFrame(data=data, + columns=['REGION', 'YEAR', 'VALUE']) - expected = pd.DataFrame(data=data, columns=['REGION', 'TIMESLICE', 'TECHNOLOGY', 'FUEL', 'YEAR', 'VALUE']) - - index = ['REGION', 'TIMESLICE', 'TECHNOLOGY', 'FUEL', 'YEAR'] + print(actual, type(actual)) + print(expected, type(expected)) - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) + pd.testing.assert_frame_equal(actual, expected) class TestEnergy: - def test_filter_ProdByTechAn(self): + def test_filter_capacity(self): filepath = os.path.join("tests","fixtures","ProductionByTechnologyAnnual.csv") input_data = pd.read_csv(filepath) technologies = ['^.{6}(I0)','^.{6}(X0)','^.{2}(HY)','^.{2}(OC)','^.{2}(SO)','^.{2}(WI)'] - actual = filter_ProdByTechAn(input_data, technologies) + actual = filter_capacity(input_data, technologies) data = [ ['AT', 2015, 26.324108350683794], @@ -118,16 +154,14 @@ def test_filter_ProdByTechAn(self): expected = pd.DataFrame(data=data, columns=["REGION", "YEAR","VALUE"]) - index = ["REGION", "YEAR"] - - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) + pd.testing.assert_frame_equal(actual, expected) def test_filter_primary_bm(self): filepath = os.path.join("tests","fixtures","ProductionByTechnologyAnnual.csv") input_data = pd.read_csv(filepath) technologies = ['(?=^.{2}(BM))^.{4}(00)','(?=^.{2}(WS))^.{4}(00)'] - actual = filter_ProdByTechAn(input_data, technologies) + actual = filter_capacity(input_data, technologies) data = [ ['AT',2015,26.324108350683794], @@ -139,16 +173,14 @@ def test_filter_primary_bm(self): expected = pd.DataFrame(data=data, columns=["REGION", "YEAR","VALUE"]) - index = ["REGION", "YEAR"] - - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) + pd.testing.assert_frame_equal(actual, expected) def test_filter_primary_co(self): filepath = os.path.join("tests","fixtures","ProductionByTechnologyAnnual.csv") input_data = pd.read_csv(filepath) technologies = ['(?=^.{2}(CO))^.{4}(00)'] - actual = filter_ProdByTechAn(input_data, technologies) + actual = filter_capacity(input_data, technologies) data = [ ['CH',2047,69.9750212433476], @@ -160,16 +192,14 @@ def test_filter_primary_co(self): expected = pd.DataFrame(data=data, columns=["REGION", "YEAR","VALUE"]) - index = ["REGION", "YEAR"] - - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) + pd.testing.assert_frame_equal(actual, expected) def test_filter_primary_ng(self): filepath = os.path.join("tests","fixtures","ProductionByTechnologyAnnual.csv") input_data = pd.read_csv(filepath) technologies = ['(?=^.{2}(NG))^.{4}(00)'] - actual = filter_ProdByTechAn(input_data, technologies) + actual = filter_capacity(input_data, technologies) data = [ ['BE',2016,141.0], @@ -177,67 +207,56 @@ def test_filter_primary_ng(self): expected = pd.DataFrame(data=data, columns=["REGION", "YEAR","VALUE"]) - index = ["REGION", "YEAR"] - - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) + pd.testing.assert_frame_equal(actual, expected) def test_filter_primary_go(self): filepath = os.path.join("tests","fixtures","ProductionByTechnologyAnnual.csv") input_data = pd.read_csv(filepath) technologies = ['(?=^.{2}(GO))^.{4}(00)'] - actual = filter_ProdByTechAn(input_data, technologies) + actual = filter_capacity(input_data, technologies) data = [ ['BG',2015,1.423512], ] expected = pd.DataFrame(data=data, columns=["REGION", "YEAR","VALUE"]) - - index = ["REGION", "YEAR"] - - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) + pd.testing.assert_frame_equal(actual, expected) def test_filter_primary_hy(self): filepath = os.path.join("tests","fixtures","ProductionByTechnologyAnnual.csv") input_data = pd.read_csv(filepath) technologies = ['^.{2}(HY)'] - actual = filter_ProdByTechAn(input_data, technologies) + actual = filter_capacity(input_data, technologies) data = [ ['CZ',2015,3.3637616987287244], ] expected = pd.DataFrame(data=data, columns=["REGION", "YEAR","VALUE"]) - - index = ["REGION", "YEAR"] - - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) + pd.testing.assert_frame_equal(actual, expected) def test_filter_primary_nu(self): filepath = os.path.join("tests","fixtures","ProductionByTechnologyAnnual.csv") input_data = pd.read_csv(filepath) technologies = ['^.{2}(UR)'] - actual = filter_ProdByTechAn(input_data, technologies) + actual = filter_capacity(input_data, technologies) data = [ ['CZ',2015,326.2313192401038], ] expected = pd.DataFrame(data=data, columns=["REGION", "YEAR","VALUE"]) - - index = ["REGION", "YEAR"] - - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) + pd.testing.assert_frame_equal(actual, expected) def test_filter_primary_oc(self): filepath = os.path.join("tests","fixtures","ProductionByTechnologyAnnual.csv") input_data = pd.read_csv(filepath) technologies = ['^.{2}(OC)'] - actual = filter_ProdByTechAn(input_data, technologies) + actual = filter_capacity(input_data, technologies) data = [ ['DK',2015,0.0031536000000000003], @@ -245,16 +264,14 @@ def test_filter_primary_oc(self): expected = pd.DataFrame(data=data, columns=["REGION", "YEAR","VALUE"]) - index = ["REGION", "YEAR"] - - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) + pd.testing.assert_frame_equal(actual, expected) def test_filter_primary_oi(self): filepath = os.path.join("tests","fixtures","ProductionByTechnologyAnnual.csv") input_data = pd.read_csv(filepath) technologies = ['(?=^.{2}(OI))^.{4}(00)','(?=^.{2}(HF))^.{4}(00)'] - actual = filter_ProdByTechAn(input_data, technologies) + actual = filter_capacity(input_data, technologies) data = [ ['EE',2015,28.512107999999998], @@ -262,16 +279,14 @@ def test_filter_primary_oi(self): expected = pd.DataFrame(data=data, columns=["REGION", "YEAR","VALUE"]) - index = ["REGION", "YEAR"] - - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) + pd.testing.assert_frame_equal(actual, expected) def test_filter_primary_so(self): filepath = os.path.join("tests","fixtures","ProductionByTechnologyAnnual.csv") input_data = pd.read_csv(filepath) technologies = ['^.{2}(SO)'] - actual = filter_ProdByTechAn(input_data, technologies) + actual = filter_capacity(input_data, technologies) data = [ ['ES',2015,26.75595496070811], @@ -279,16 +294,14 @@ def test_filter_primary_so(self): expected = pd.DataFrame(data=data, columns=["REGION", "YEAR","VALUE"]) - index = ["REGION", "YEAR"] - - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) + pd.testing.assert_frame_equal(actual, expected) def test_filter_primary_wi(self): filepath = os.path.join("tests","fixtures","ProductionByTechnologyAnnual.csv") input_data = pd.read_csv(filepath) technologies = ['^.{2}(WI)'] - actual = filter_ProdByTechAn(input_data, technologies) + actual = filter_capacity(input_data, technologies) data = [ ['FI', 2015, 0.29658110158442175], @@ -297,16 +310,14 @@ def test_filter_primary_wi(self): expected = pd.DataFrame(data=data, columns=["REGION", "YEAR","VALUE"]) - index = ["REGION", "YEAR"] - - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) + pd.testing.assert_frame_equal(actual, expected) def test_filter_secondary_bm(self): filepath = os.path.join("tests","fixtures","ProductionByTechnologyAnnual.csv") input_data = pd.read_csv(filepath) technologies = ['(?=^.{2}(BF))^((?!00).)*$','(?=^.{2}(BM))^((?!00).)*$', '(?=^.{2}(WS))^((?!00).)*$'] - actual = filter_ProdByTechAn(input_data, technologies) + actual = filter_capacity(input_data, technologies) data = [ ['AT', 2042, 0.6636346353894057], @@ -318,9 +329,7 @@ def test_filter_secondary_bm(self): expected = pd.DataFrame(data=data, columns=["REGION", "YEAR","VALUE"]) - index = ["REGION", "YEAR"] - - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) + pd.testing.assert_frame_equal(actual, expected) def test_filter_final_energy(self): filepath = os.path.join("tests","fixtures","Demand.csv") @@ -336,9 +345,10 @@ def test_filter_final_energy(self): expected = pd.DataFrame(data=data, columns=["REGION", "YEAR", "VALUE"]) - index = ["REGION", "YEAR"] + print(actual) + print(expected) - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) + pd.testing.assert_frame_equal(actual, expected) def test_fuels_format_wrong(self): filepath = os.path.join("tests","fixtures","Demand.csv") @@ -374,9 +384,11 @@ def test_filter_inst_capacity(self): expected = pd.DataFrame(data=data, columns=["REGION", "YEAR", "VALUE"]) - index = ["REGION", "YEAR"] + print(actual) - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) + print(expected) + + pd.testing.assert_frame_equal(actual, expected) def test_filter_inst_capacity_bio(self): filepath = os.path.join("tests","fixtures","TotalCapacityAnnual.csv") @@ -392,9 +404,7 @@ def test_filter_inst_capacity_bio(self): expected = pd.DataFrame(data=data, columns=["REGION", "YEAR", "VALUE"]) - index = ["REGION", "YEAR"] - - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) + pd.testing.assert_frame_equal(actual, expected) def test_filter_inst_capacity_coal(self): filepath = os.path.join("tests","fixtures","TotalCapacityAnnual.csv") @@ -408,9 +418,7 @@ def test_filter_inst_capacity_coal(self): expected = pd.DataFrame(data=data, columns=["REGION", "YEAR", "VALUE"]) - index = ["REGION", "YEAR"] - - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) + pd.testing.assert_frame_equal(actual, expected) def test_filter_inst_capacity_gas(self): filepath = os.path.join("tests","fixtures","TotalCapacityAnnual.csv") @@ -424,9 +432,7 @@ def test_filter_inst_capacity_gas(self): expected = pd.DataFrame(data=data, columns=["REGION", "YEAR", "VALUE"]) - index = ["REGION", "YEAR"] - - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) + pd.testing.assert_frame_equal(actual, expected) def test_filter_inst_capacity_geo(self): filepath = os.path.join("tests","fixtures","TotalCapacityAnnual.csv") @@ -440,9 +446,7 @@ def test_filter_inst_capacity_geo(self): expected = pd.DataFrame(data=data, columns=["REGION", "YEAR", "VALUE"]) - index = ["REGION", "YEAR"] - - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) + pd.testing.assert_frame_equal(actual, expected) def test_filter_inst_capacity_hydro(self): filepath = os.path.join("tests","fixtures","TotalCapacityAnnual.csv") @@ -456,9 +460,7 @@ def test_filter_inst_capacity_hydro(self): expected = pd.DataFrame(data=data, columns=["REGION", "YEAR", "VALUE"]) - index = ["REGION", "YEAR"] - - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) + pd.testing.assert_frame_equal(actual, expected) def test_filter_inst_capacity_nuclear(self): filepath = os.path.join("tests","fixtures","TotalCapacityAnnual.csv") @@ -472,9 +474,7 @@ def test_filter_inst_capacity_nuclear(self): expected = pd.DataFrame(data=data, columns=["REGION", "YEAR", "VALUE"]) - index = ["REGION", "YEAR"] - - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) + pd.testing.assert_frame_equal(actual, expected) def test_filter_inst_capacity_ocean(self): filepath = os.path.join("tests","fixtures","TotalCapacityAnnual.csv") @@ -488,9 +488,7 @@ def test_filter_inst_capacity_ocean(self): expected = pd.DataFrame(data=data, columns=["REGION", "YEAR", "VALUE"]) - index = ["REGION", "YEAR"] - - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) + pd.testing.assert_frame_equal(actual, expected) def test_filter_inst_capacity_oil(self): filepath = os.path.join("tests","fixtures","TotalCapacityAnnual.csv") @@ -504,9 +502,7 @@ def test_filter_inst_capacity_oil(self): expected = pd.DataFrame(data=data, columns=["REGION", "YEAR", "VALUE"]) - index = ["REGION", "YEAR"] - - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) + pd.testing.assert_frame_equal(actual, expected) def test_filter_inst_capacity_solar(self): filepath = os.path.join("tests","fixtures","TotalCapacityAnnual.csv") @@ -520,9 +516,7 @@ def test_filter_inst_capacity_solar(self): expected = pd.DataFrame(data=data, columns=["REGION", "YEAR", "VALUE"]) - index = ["REGION", "YEAR"] - - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) + pd.testing.assert_frame_equal(actual, expected) def test_filter_inst_capacity_wi_offshore(self): filepath = os.path.join("tests","fixtures","TotalCapacityAnnual.csv") @@ -536,9 +530,7 @@ def test_filter_inst_capacity_wi_offshore(self): expected = pd.DataFrame(data=data, columns=["REGION", "YEAR", "VALUE"]) - index = ["REGION", "YEAR"] - - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) + pd.testing.assert_frame_equal(actual, expected) class TestPrice: @@ -546,17 +538,18 @@ def test_price_bm(self): filepath = os.path.join("tests","fixtures","VariableCost.csv") input_data = pd.read_csv(filepath) commodity = ['(?=^.{2}(BM))^.{6}(X0)'] - actual = filter_var_cost(input_data, commodity) + actual = filter_capacity(input_data, commodity) data = [ - ['AT',2015,2.327755063], - ['AT',2016,2.327755063], - ['BE',2015,2.327755063], - ['BE',2016,2.327755063], + ['AT',2015,3.0], + ['AT',2016,4.0], + ['BE',2015,1.7], + ['BE',2016,1.8], ] expected = pd.DataFrame(data=data, columns=["REGION", "YEAR", "VALUE"]) - index = ["REGION", "YEAR"] + print(actual) + print(expected) - pd.testing.assert_frame_equal(actual.set_index(index), expected.set_index(index), check_index_type=False) \ No newline at end of file + pd.testing.assert_frame_equal(actual, expected)