Updated Snakemake Formatting

[trevorb1]

Updated the snakefile to match the documentation suggested standards (ie. it now passes snakemake --lint), and removed any non-used import statements from the modules

Included in the update:

• Added logging
• Added conda environments (see here for more info)
• Fixed path concatenation

[trevorb1]

There is a small rounding difference happening in the residualCapacity.py file happening now. About 50 techs have different values at the 3 decimal place (see sample below). I updated a bunch of my packages so I am thinking its just something to due with that?

ResidualCapacity.csv
The two files are different.

Deleted Elements:
NAmerica PWRCCGUSAFL01 2019 36.284
NAmerica PWRCTGUSATX01 2035 14.34
NAmerica PWRCOAUSACA01 2031 0.006
NAmerica PWRCOAUSACA01 2049 0.002
NAmerica PWRHYDUSASW01 2026 2.79
NAmerica PWRHYDUSAFL01 2019 0.044
...

Added Elements:
NAmerica PWRCCGUSAFL01 2019 36.285
NAmerica PWRCTGUSATX01 2035 14.341
NAmerica PWRCOAUSACA01 2031 0.005
NAmerica PWRCOAUSACA01 2049 0.003
NAmerica PWRHYDUSASW01 2026 2.791
NAmerica PWRHYDUSAFL01 2019 0.043
...

[Kiel-SFU]

According to https://snakemake.readthedocs.io/en/stable/snakefiles/deployment.html, Snakemake recommends a particular file organization structure (for example, the snakefile and envs are in a folder called 'workflow'). Do you think we should structure the codebase more closely to this, or even in the exact suggested way?

[trevorb1]

Good point, and its part of the update coming !

[Kiel-SFU]

Not sure why openpyxl, ca-certificates, openssl, pyyaml are necessary here?

[trevorb1]

Those are just the dependencies it spat out at me that I needed. I started with a fresh environment and only added packages when it asked to try and get the bare minimum. pyyaml was needed to get past reading from the config.yaml file (not sure how its different from regular yaml tbh). Not sure about ca-certificates and openssl. Does it run without them?

[Kiel-SFU]

Hmm... could you check in the error message to see if it says where the dependencies are needed? They don't seem to be required for me when I remove them and run snakemake all

Edit: Never mind, these are required as you pointed out.

[trevorb1]

your using the --use-conda flag as well, right?

[Kiel-SFU]

As it is, it looks like the model will throw an error if someone has a more recent (>0.11.0) otoole version. If exact otoole version 0.11.0 is recommended to generate the data for the Canada-US model, it should probably be specified in a README somewhere as well.

[trevorb1]

Is there a newer version then 0.11.0? I think I (successfully) tested it on 0.10.0 as well, cause there is sometime backwards compatibility issues. Nevertheless, good idea! Did you want to add it in?

[Kiel-SFU]

It does seem to work with otoole 0.10.0, and I don't know why. It's odd that we'd specify a version if it works with older versions anyways, but it's functional, so I think we can leave it be, if you think it's okay.

[trevorb1]

Its just for when you create the conda environment using the --use-conda flag

[Kiel-SFU]

The changes look good/interesting. I've left some questions throughout where I think changes could maybe be made, but it all runs properly.