Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add BB4CMIP processing #27

Merged
merged 13 commits into from
Jan 14, 2025
Merged

Add BB4CMIP processing #27

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
a4fbfac
Add first step of BB4CMIP processing
znichollscr Jan 8, 2025
15a9c1f
Clean up BB4CMIP processing and add post-processing
znichollscr Jan 9, 2025
c2463cf
Complete run through except NMVOC
znichollscr Jan 9, 2025
7929bf5
Update BB4CMIP download
znichollscr Jan 9, 2025
007fe19
Add NMVOC into BB4CMIP processing
znichollscr Jan 9, 2025
4ee8aac
Add unit conversion to BB4CMIP output
znichollscr Jan 9, 2025
745074e
Add check that regional sum == global
znichollscr Jan 9, 2025
a323f1c
Clean up afer rebase
znichollscr Jan 10, 2025
f98359f
Remove leftovers
znichollscr Jan 10, 2025
1c8d1ec
Clean up
znichollscr Jan 10, 2025
37a0bb7
Last piece of clean up
znichollscr Jan 10, 2025
9e72ed2
One more bit of clean up
znichollscr Jan 10, 2025
4805728
Update notebook slightly
znichollscr Jan 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions data/national/gfed-bb4cmip/data_raw/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,12 @@ Then, download the data
```sh
# Comma-separated list
# You can search with esgpull search e.g. `venv/bin/esgpull search --all project:input4MIPs mip_era:CMIP6Plus source_id:DRES-CMIP-BB4CMIP7-1-0 grid_label:gn`
variable_ids_to_grab="BC,CH4,CO,CO2,N2O,NH3,NOx,OC,SO2,gridcellarea"
variable_ids_to_grab="BC,CH4,CO,CO2,Multiple,N2O,NH3,NMVOC_bulk,NOx,OC,SO2,gridcellarea"
venv/bin/esgpull add --tag bb4cmip --track variable_id:"${variable_ids_to_grab}" project:input4MIPs mip_era:CMIP6Plus source_id:DRES-CMIP-BB4CMIP7-1-0 grid_label:gn
venv/bin/esgpull update -y --tag bb4cmip
venv/bin/esgpull download
venv/bin/esgpull download --tag bb4cmip

venv/bin/esgpull add --tag bb4cmip-nmvoc-bulk --track variable_long_name:"biomass_burning_NMVOC_bulk_flux" project:input4MIPs mip_era:CMIP6Plus source_id:DRES-CMIP-BB4CMIP7-1-0 grid_label:gn
venv/bin/esgpull update -y --tag bb4cmip-nmvoc-bulk
venv/bin/esgpull download --tag bb4cmip-nmvoc-bulk
```
39 changes: 11 additions & 28 deletions notebooks/0101_CEDS-prepare.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
CEDS_PROCESSING_ID,
DATA_ROOT,
)
from emissions_harmonization_historical.units import assert_units_match_wishes

# set unit registry
pix.units.set_openscm_registry_as_default()
Expand Down Expand Up @@ -105,9 +106,12 @@
# %%
ceds

# %%
ceds

# %%
# change units to align with IAM scenario data
# adjust units; change column 'units' to 'units' and add '/yr'
# adjust units; change column 'units' to 'unit' and add '/yr'
ceds = ceds.pix.dropna(subset=["units"]).pix.format(unit="{units}/yr", drop=True)
# adjust units; change all to values to Mt instead of kt
ceds = pix.units.convert_unit(ceds, lambda x: "Mt " + x.removeprefix("kt").strip())
Expand Down Expand Up @@ -157,31 +161,6 @@
ceds_reformatted = ceds.rename_axis(index={"em": "variable", "country": "region"})
ceds_reformatted

# %%
# check that all units are as wished.
# ... as this is for the harmonization of IAMs (in CMIP7 / ScenarioMIP), the desired units are defined in https://github.com/IAMconsortium/common-definitions/
unit_wishes = pd.MultiIndex.from_tuples(
[
("BC", "Mt BC/yr"),
("CH4", "Mt CH4/yr"),
("N2O", "kt N2O/yr"),
("CO", "Mt CO/yr"),
("CO2", "Mt CO2/yr"),
("NH3", "Mt NH3/yr"),
("NMVOC", "Mt NMVOC/yr"),
("NOx", "Mt NO2/yr"),
("OC", "Mt OC/yr"),
("Sulfur", "Mt SO2/yr"),
],
names=["variable", "unit"],
)

# check that we follow the desired units
ceds_reformatted.pix.unique(unit_wishes.names)
unit_mismatches = ceds_reformatted.pix.unique(unit_wishes.names).symmetric_difference(unit_wishes)

assert unit_mismatches.empty, f"Unit mismatches detected:\n{unit_mismatches}"

# %%
# rename to IAMC-style variable names including standard index order
ceds_reformatted_iamc = (
Expand All @@ -191,6 +170,12 @@
).sort_values(by=["region", "variable"])
ceds_reformatted_iamc

# %%
ceds_reformatted_iamc

# %%
assert_units_match_wishes(ceds_reformatted_iamc)

# %% [markdown]
# Save formatted CEDS data

Expand Down Expand Up @@ -243,7 +228,6 @@
.set_index(out_global.index.names)
)
national_sums_checker.columns = national_sums_checker.columns.astype(int)
national_sums_checker
pd.testing.assert_frame_equal(global_checker, national_sums_checker, check_like=True)

# check that aircraft and international shipping are zero:
Expand All @@ -269,7 +253,6 @@
== CEDS_EXPECTED_NUMBER_OF_REGION_VARIABLE_PAIRS_IN_GLOBAL_HARMONIZATION
) # ... if the CEDS version is not 2024_07_08, this assert statement may need to be updated


# %%
# national CEDS data
ceds_processed_output_file_national.parent.mkdir(exist_ok=True, parents=True)
Expand Down
Loading