CMIP7 derived variables

[lewisjared]

Description

ESMValCore cannot load CMOR table definitions for the CMIP7 project for derived variables.
When running recipes with CMIP7 datasets that include derived variables, for example equilibrium-climate-sensitivity, The variable lookup fails with:

Unable to load CMOR table (project) 'CMIP7' for variable '<var>' with mip '<realm>'

Other native variables worked and this only impacted derived variables

How to reproduce

Run any ESMValTool recipe that includes a dataset with project: CMIP7. The recipe will fail at variable validation when ESMValCore tries to look up the CMOR table entry.

Examples of failing variable lookups:

• Unable to load CMOR table (project) 'CMIP7' for variable 'lwcre' with mip 'atmos'
• Unable to load CMOR table (project) 'CMIP7' for variable 'swcre' with mip 'atmos'
• Unable to load CMOR table (project) 'CMIP7' for variable 'rtnt' with mip 'atmos'

Log output from uv run ref test-cases run --provider esmvaltool --test-case cmip7 --diagnostic cloud-radiative-effects --fetch

2026-02-15 16:15:48.318 +11:00 | INFO     | climate_ref.database - Connecting to database at sqlite:////Users/jared/code/climate-ref/climate-ref-worktrees/cmip7-esmvaltool/.ref/db/climate_ref.db
2026-02-15 16:15:48.325 +11:00 | INFO     | climate_ref.database - Creating backup of database at /Users/jared/code/climate-ref/climate-ref-worktrees/cmip7-esmvaltool/.ref/db/backups/climate_ref_20260215_161548.db
2026-02-15 16:15:48.327 +11:00 | INFO     | climate_ref.database - Removing old backup /Users/jared/code/climate-ref/climate-ref-worktrees/cmip7-esmvaltool/.ref/db/backups/climate_ref_20260215_131108.db
2026-02-15 16:15:49.359 +11:00 | INFO     | climate_ref.cli.test_cases - Found 1 test case(s) to run
2026-02-15 16:15:49.360 +11:00 | INFO     | climate_ref.cli.test_cases - Fetching test data for esmvaltool/cloud-radiative-effects/cmip7
2026-02-15 16:15:49.423 +11:00 | INFO     | climate_ref_core.esgf.fetcher - Fetching datasets for request: cmip7
2026-02-15 16:15:49.425 +11:00 | INFO     | intake_esgf.logging - �[36;32msearch begin�[0m experiment_id=['historical'], source_id=['CanESM5'], variable_id=['areacella', 'rlut', 'rlutcs', 'rsut', 'rsutcs'], member_id=['r1i1p1f1'], frequency=['fx', 'mon'], type=['Dataset'], project=['CMIP6'], latest=[True], retracted=[False]

   Searching indices:   0%|          |0/1 [       ?index/s]2026-02-15 16:15:49.446 +11:00 | INFO     | intake_esgf.logging - └─GlobusESGFIndex('ESGF2-US-1.5-Catalog') results=20 response_time=0.01

   Searching indices: 100%|██████████|1/1 [  124.88index/s]
2026-02-15 16:15:49.452 +11:00 | INFO     | intake_esgf.logging - combine_time=0.00
2026-02-15 16:15:49.453 +11:00 | INFO     | intake_esgf.logging - �[36;32msearch end�[0m total_time=0.03
2026-02-15 16:15:49.457 +11:00 | INFO     | intake_esgf.logging - �[36;32mfile info begin�[0m
2026-02-15 16:15:49.460 +11:00 | INFO     | intake_esgf.logging - └─GlobusESGFIndex('ESGF2-US-1.5-Catalog') results=20 response_time=0.00
2026-02-15 16:15:49.460 +11:00 | INFO     | intake_esgf.logging - combine_time=0.00
2026-02-15 16:15:49.461 +11:00 | INFO     | intake_esgf.logging - �[36;32mfile info end�[0m total_time=0.00
2026-02-15 16:15:49.466 +11:00 | INFO     | climate_ref_core.esgf.cmip7 - Converting to CMIP7: rlut_Amon_CanESM5_historical_r1i1p1f1_gn_185001-201412.nc
2026-02-15 16:15:49.516 +11:00 | INFO     | climate_ref_core.esgf.cmip7 - Converting to CMIP7: rlutcs_Amon_CanESM5_historical_r1i1p1f1_gn_185001-201412.nc
2026-02-15 16:15:49.551 +11:00 | INFO     | climate_ref_core.esgf.cmip7 - Converting to CMIP7: rsut_Amon_CanESM5_historical_r1i1p1f1_gn_185001-201412.nc
2026-02-15 16:15:49.587 +11:00 | INFO     | climate_ref_core.esgf.cmip7 - Converting to CMIP7: rsutcs_Amon_CanESM5_historical_r1i1p1f1_gn_185001-201412.nc
2026-02-15 16:15:49.622 +11:00 | INFO     | climate_ref_core.esgf.cmip7 - Converting to CMIP7: areacella_fx_CanESM5_historical_r1i1p1f1_gn.nc
2026-02-15 16:15:49.624 +11:00 | INFO     | climate_ref_core.esgf.fetcher - Found 5 datasets for request: cmip7
2026-02-15 16:15:49.625 +11:00 | INFO     | climate_ref_core.esgf.fetcher - Fetched 5 files for request: cmip7
2026-02-15 16:15:49.690 +11:00 | INFO     | climate_ref_core.testing - Saved catalog to /Users/jared/code/climate-ref/climate-ref-worktrees/cmip7-esmvaltool/packages/climate-ref-esmvaltool/tests/test-data/cloud-radiative-effects/cmip7/catalog.yaml (paths: /Users/jared/code/climate-ref/climate-ref-worktrees/cmip7-esmvaltool/packages/climate-ref-esmvaltool/tests/test-data/cloud-radiative-effects/cmip7/catalog.paths.yaml)
2026-02-15 16:15:49.690 +11:00 | INFO     | climate_ref.cli.test_cases - Running test case 'cmip7' for esmvaltool/cloud-radiative-effects
2026-02-15 16:15:49.695 +11:00 | INFO     | climate_ref_esmvaltool.diagnostics.base - Using ESMValTool recipe:
documentation:
  title: Model evaluation with focus on the cloud radiative effect.
  description: 'Plot climatologies and zonal mean profiles of cloud radiative effects
    (sw + lw) for the REF.

    '
  authors:
  - bock_lisa
  - lauer_axel
  maintainer:
  - bock_lisa
datasets:
- project: CMIP7
  activity: CMIP
  branding_suffix: tavg-u-hxy-u
  dataset: CanESM5
  ensemble: r1i1p1f1
  exp: historical
  frequency: mon
  grid: gn
  institute: CCCma
  mip: atmos
  region: glb
timerange_for_models:
  timerange: 1996/2014
preprocessors:
  full_climatology:
    climate_statistics:
      period: full
    regrid:
      target_grid: 1x1
      scheme:
        reference: esmf_regrid.schemes:ESMFAreaWeighted
  zonal_mean:
    climate_statistics:
      period: full
    regrid:
      scheme:
        reference: esmf_regrid.schemes:ESMFAreaWeighted
      target_grid: 1x1
    zonal_statistics:
      operator: mean
diagnostics:
  plot_maps:
    description: Plot lwcre climatology maps including reference datasets.
    variables:
      lwcre:
        timerange: 1996/2014
        mip: atmos
        preprocessor: full_climatology
        derive: true
      swcre:
        timerange: 1996/2014
        mip: atmos
        preprocessor: full_climatology
        derive: true
    additional_datasets:
    - dataset: CERES-EBAF
      project: OBS
      type: sat
      version: Ed4.2
      tier: 2
      start_year: 2000
      end_year: 2023
      reference_for_monitor_diags: true
    scripts:
      plot:
        script: monitor/multi_datasets.py
        plot_folder: '{plot_dir}'
        plot_filename: '{plot_type}_{real_name}_{dataset}_{mip}'
        plots:
          map:
            common_cbar: true
            x_pos_stats_avg: -0.1
            fontsize: 10
  plot_profiles:
    description: Plot lwcre zonal mean including reference datasets.
    variables:
      lwcre:
        timerange: 1996/2014
        mip: atmos
        preprocessor: zonal_mean
        derive: true
      swcre:
        timerange: 1996/2014
        mip: atmos
        preprocessor: zonal_mean
        derive: true
    additional_datasets:
    - dataset: CERES-EBAF
      project: OBS
      type: sat
      version: Ed4.2
      tier: 2
      start_year: 2000
      end_year: 2023
      reference_for_monitor_diags: true
    - dataset: ESACCI-CLOUD
      project: OBS
      type: sat
      version: AVHRR-AMPM-fv3.0
      tier: 2
      start_year: 1996
      end_year: 2016
    - dataset: ISCCP-FH
      project: OBS
      type: sat
      version: v0
      tier: 2
      start_year: 1996
      end_year: 2016
    scripts:
      plot:
        script: monitor/multi_datasets.py
        plot_folder: '{plot_dir}'
        plot_filename: '{plot_type}_{real_name}_{dataset}_{mip}'
        plots:
          variable_vs_lat:
            legend_kwargs:
              loc: upper right
            plot_kwargs:
              CERES-EBAF:
                color: black
              ESACCI-CLOUD:
                color: black
                linestyle: dashed
              ISCCP-FH:
                color: black
                linestyle: dotted

2026-02-15 16:15:49.698 +11:00 | INFO     | climate_ref_esmvaltool.diagnostics.base - Using ESMValTool configuration:
output_dir: /Users/jared/code/climate-ref/climate-ref-worktrees/cmip7-esmvaltool/.ref/results/test-cases/esmvaltool/cloud-radiative-effects/cmip7/executions
projects:
  CMIP6:
    data:
      local:
        dirname_template: '{project}/{activity}/{institute}/{dataset}/{exp}/{ensemble}/{mip}/{short_name}/{grid}/{version}'
        filename_template: '{short_name}_{mip}_{dataset}_{exp}_{ensemble}_{grid}*.nc'
        rootpath: /Users/jared/code/climate-ref/climate-ref-worktrees/cmip7-esmvaltool/.ref/results/test-cases/esmvaltool/cloud-radiative-effects/cmip7/climate_data
        type: esmvalcore.io.local.LocalDataSource
  CMIP7:
    data:
      local:
        dirname_template: '{project}/{activity}/{institute}/{dataset}/{exp}/{ensemble}/{region}/{frequency}/{short_name}/{branding_suffix}/{grid}/{version}'
        filename_template: '{short_name}_{branding_suffix}_{frequency}_{region}_{grid}_{dataset}_{exp}_{ensemble}*.nc'
        rootpath: /Users/jared/code/climate-ref/climate-ref-worktrees/cmip7-esmvaltool/.ref/results/test-cases/esmvaltool/cloud-radiative-effects/cmip7/climate_data
        type: esmvalcore.io.local.LocalDataSource
  OBS:
    data:
      local:
        dirname_template: Tier{tier}/{dataset}
        filename_template: '{project}_{dataset}_{type}_{version}_{mip}_{short_name}[_.]*nc'
        rootpath: /Users/jared/Library/Caches/climate_ref/ESMValTool/OBS
        type: esmvalcore.io.local.LocalDataSource
  OBS6:
    data:
      local:
        dirname_template: Tier{tier}/{dataset}
        filename_template: '{project}_{dataset}_{type}_{version}_{mip}_{short_name}[_.]*nc'
        rootpath: /Users/jared/Library/Caches/climate_ref/ESMValTool/OBS
        type: esmvalcore.io.local.LocalDataSource
  native6:
    data:
      local:
        dirname_template: Tier{tier}/{dataset}/{version}/{frequency}/{short_name}
        filename_template: '*.nc'
        rootpath: /Users/jared/Library/Caches/climate_ref/ESMValTool/native6
        type: esmvalcore.io.local.LocalDataSource
  obs4MIPs:
    data:
      esmvaltool:
        dirname_template: '{project}/{dataset}/{version}'
        filename_template: '{short_name}_*.nc'
        rootpath: /Users/jared/Library/Caches/climate_ref/ESMValTool
        type: esmvalcore.io.local.LocalDataSource
      local:
        dirname_template: '{project}/{dataset}/{version}'
        filename_template: '{short_name}_*.nc'
        rootpath: /Users/jared/code/climate-ref/climate-ref-worktrees/cmip7-esmvaltool/.ref/results/test-cases/esmvaltool/cloud-radiative-effects/cmip7/climate_data
        type: esmvalcore.io.local.LocalDataSource
search_data: quick

2026-02-15 16:15:49.699 +11:00 | INFO     | climate_ref_core.providers - Running '/Users/jared/code/climate-ref/software/conda/micromamba run --prefix /Users/jared/code/climate-ref/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308 esmvaltool run --config-dir=/Users/jared/code/climate-ref/climate-ref-worktrees/cmip7-esmvaltool/.ref/results/test-cases/esmvaltool/cloud-radiative-effects/cmip7/config /Users/jared/code/climate-ref/climate-ref-worktrees/cmip7-esmvaltool/.ref/results/test-cases/esmvaltool/cloud-radiative-effects/cmip7/recipe.yml'
2026-02-15 16:15:53.907 +11:00 | ERROR    | climate_ref_core.providers - Failed to run ['/Users/jared/code/climate-ref/software/conda/micromamba', 'run', '--prefix', '/Users/jared/code/climate-ref/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308', 'esmvaltool', 'run', '--config-dir=/Users/jared/code/climate-ref/climate-ref-worktrees/cmip7-esmvaltool/.ref/results/test-cases/esmvaltool/cloud-radiative-effects/cmip7/config', '/Users/jared/code/climate-ref/climate-ref-worktrees/cmip7-esmvaltool/.ref/results/test-cases/esmvaltool/cloud-radiative-effects/cmip7/recipe.yml']
2026-02-15 16:15:53.908 +11:00 | ERROR    | climate_ref_core.providers - 2026-02-15 05:15:52,646 UTC [1906] INFO    
______________________________________________________________________
          _____ ____  __  ____     __    _ _____           _
         | ____/ ___||  \/  \ \   / /_ _| |_   _|__   ___ | |
         |  _| \___ \| |\/| |\ \ / / _` | | | |/ _ \ / _ \| |
         | |___ ___) | |  | | \ V / (_| | | | | (_) | (_) | |
         |_____|____/|_|  |_|  \_/ \__,_|_| |_|\___/ \___/|_|
______________________________________________________________________

Earth System Model Evaluation Tool

A community tool for the evaluation of Earth system models.

https://esmvaltool.org

The Earth System Model Evaluation Tool (ESMValTool) is a community
diagnostics and performance metrics tool for the evaluation of Earth
System Models (ESMs) that allows for routine comparison of single or
multiple models, either against predecessor versions or against
observations.

Tutorial: https://tutorial.esmvaltool.org
Documentation: https://docs.esmvaltool.org
Contact: esmvaltool-dev@listserv.dfn.de

If you find this software useful for your research, please cite it using
https://doi.org/10.5281/zenodo.3387139 for ESMValCore or
https://doi.org/10.5281/zenodo.3401363 for ESMValTool or
any of the reference papers listed at https://esmvaltool.org/references/.

Have fun!

2026-02-15 05:15:52,646 UTC [1906] INFO    Package versions
2026-02-15 05:15:52,646 UTC [1906] INFO    ----------------
2026-02-15 05:15:52,646 UTC [1906] INFO    ESMValCore: 2.14.0rc1
2026-02-15 05:15:52,646 UTC [1906] INFO    ESMValTool: 2.14.0.dev65+gf5214c924
2026-02-15 05:15:52,646 UTC [1906] INFO    ----------------
2026-02-15 05:15:52,646 UTC [1906] INFO    Reading configuration files from:
/Users/jared/code/climate-ref/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/config/configurations/defaults (defaults)
/Users/jared/.config/esmvaltool (default user configuration directory)
/Users/jared/code/climate-ref/climate-ref-worktrees/cmip7-esmvaltool/.ref/results/test-cases/esmvaltool/cloud-radiative-effects/cmip7/config (command line argument)
2026-02-15 05:15:52,646 UTC [1906] INFO    Writing program log files to:
/Users/jared/code/climate-ref/climate-ref-worktrees/cmip7-esmvaltool/.ref/results/test-cases/esmvaltool/cloud-radiative-effects/cmip7/executions/recipe_20260215_051552/run/main_log.txt
/Users/jared/code/climate-ref/climate-ref-worktrees/cmip7-esmvaltool/.ref/results/test-cases/esmvaltool/cloud-radiative-effects/cmip7/executions/recipe_20260215_051552/run/main_log_debug.txt
/Users/jared/code/climate-ref/climate-ref-worktrees/cmip7-esmvaltool/.ref/results/test-cases/esmvaltool/cloud-radiative-effects/cmip7/executions/recipe_20260215_051552/run/cmor_log.txt
2026-02-15 05:15:53,011 UTC [1906] INFO    Starting the Earth System Model Evaluation Tool at time: 2026-02-15 05:15:53 UTC
2026-02-15 05:15:53,011 UTC [1906] INFO    ----------------------------------------------------------------------
2026-02-15 05:15:53,011 UTC [1906] INFO    RECIPE   = /Users/jared/code/climate-ref/climate-ref-worktrees/cmip7-esmvaltool/.ref/results/test-cases/esmvaltool/cloud-radiative-effects/cmip7/recipe.yml
2026-02-15 05:15:53,011 UTC [1906] INFO    RUNDIR     = /Users/jared/code/climate-ref/climate-ref-worktrees/cmip7-esmvaltool/.ref/results/test-cases/esmvaltool/cloud-radiative-effects/cmip7/executions/recipe_20260215_051552/run
2026-02-15 05:15:53,011 UTC [1906] INFO    WORKDIR    = /Users/jared/code/climate-ref/climate-ref-worktrees/cmip7-esmvaltool/.ref/results/test-cases/esmvaltool/cloud-radiative-effects/cmip7/executions/recipe_20260215_051552/work
2026-02-15 05:15:53,011 UTC [1906] INFO    PREPROCDIR = /Users/jared/code/climate-ref/climate-ref-worktrees/cmip7-esmvaltool/.ref/results/test-cases/esmvaltool/cloud-radiative-effects/cmip7/executions/recipe_20260215_051552/preproc
2026-02-15 05:15:53,011 UTC [1906] INFO    PLOTDIR    = /Users/jared/code/climate-ref/climate-ref-worktrees/cmip7-esmvaltool/.ref/results/test-cases/esmvaltool/cloud-radiative-effects/cmip7/executions/recipe_20260215_051552/plots
2026-02-15 05:15:53,011 UTC [1906] INFO    ----------------------------------------------------------------------
2026-02-15 05:15:53,039 UTC [1906] INFO    Running tasks using at most 2 processes
2026-02-15 05:15:53,039 UTC [1906] INFO    If your system hangs during execution, it may not have enough memory for keeping this number of tasks in memory.
2026-02-15 05:15:53,039 UTC [1906] INFO    If you experience memory problems, try reducing 'max_parallel_tasks' in your configuration.
2026-02-15 05:15:53,650 UTC [1906] INFO    Maximum memory used (estimate): 0.0 GB
2026-02-15 05:15:53,651 UTC [1906] INFO    Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur.
2026-02-15 05:15:53,651 UTC [1906] ERROR   Unable to load CMOR table (project) 'CMIP7' for variable 'lwcre' with mip 'atmos'

2026-02-15 16:15:53.908 +11:00 | ERROR    | climate_ref.cli.test_cases - Diagnostic execution failed for esmvaltool/cloud-radiative-effects/cmip7: Command '['/Users/jared/code/climate-ref/software/conda/micromamba', 'run', '--prefix', '/Users/jared/code/climate-ref/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308', 'esmvaltool', 'run', '--config-dir=/Users/jared/code/climate-ref/climate-ref-worktrees/cmip7-esmvaltool/.ref/results/test-cases/esmvaltool/cloud-radiative-effects/cmip7/config', '/Users/jared/code/climate-ref/climate-ref-worktrees/cmip7-esmvaltool/.ref/results/test-cases/esmvaltool/cloud-radiative-effects/cmip7/recipe.yml']' returned non-zero exit status 1.

Results: 0 passed, 1 failed
Failed test cases:
  - esmvaltool/cloud-radiative-effects/cmip7

Context

This was discovered while running CMIP7 test cases for Climate REF. The data loads correctly but recipe execution fails at variable validation.

It affects 3 of 16 CMIP7 test cases entirely:

• cloud-radiative-effects (derived lwcre/atmos)
• cloud-scatterplots-clt-swcre (derived swcre/atmos)
• equilibrium-climate-sensitivity (derived rtnt/atmos)

Environment

• ESMValCore: commit da81d5f67158f3d2603831b56ab6b4fb8a388d86

[bouweandela]

Thanks for reporting @lewisjared! I will take a look today.

[bouweandela]

The following will probably not solve this issue, but it should take us one step closer to a working recipe:

Looking at the recipe, I think it will not work as it is, because for the datasets from the OBS project, the CMIP5 CMOR tables are used and therefore a CMIP5 value for mip is expected. Could you try the following recipe:

documentation:
  title: Model evaluation with focus on the cloud radiative effect.
  description: 'Plot climatologies and zonal mean profiles of cloud radiative effects
    (sw + lw) for the REF.

    '
  authors:
  - bock_lisa
  - lauer_axel
  maintainer:
  - bock_lisa
datasets:
- project: CMIP7
  activity: CMIP
  branding_suffix: tavg-u-hxy-u
  dataset: CanESM5
  ensemble: r1i1p1f1
  exp: historical
  frequency: mon
  grid: gn
  institute: CCCma
  mip: atmos
  region: glb
timerange_for_models:
  timerange: 1996/2014
preprocessors:
  full_climatology:
    climate_statistics:
      period: full
    regrid:
      target_grid: 1x1
      scheme:
        reference: esmf_regrid.schemes:ESMFAreaWeighted
  zonal_mean:
    climate_statistics:
      period: full
    regrid:
      scheme:
        reference: esmf_regrid.schemes:ESMFAreaWeighted
      target_grid: 1x1
    zonal_statistics:
      operator: mean
diagnostics:
  plot_maps:
    description: Plot lwcre climatology maps including reference datasets.
    variables:
      lwcre:
        timerange: 1996/2014
        mip: Amon
        preprocessor: full_climatology
        derive: true
      swcre:
        timerange: 1996/2014
        mip: Amon
        preprocessor: full_climatology
        derive: true
    additional_datasets:
    - dataset: CERES-EBAF
      project: OBS
      type: sat
      version: Ed4.2
      tier: 2
      start_year: 2000
      end_year: 2023
      reference_for_monitor_diags: true
    scripts:
      plot:
        script: monitor/multi_datasets.py
        plot_folder: '{plot_dir}'
        plot_filename: '{plot_type}_{real_name}_{dataset}_{mip}'
        plots:
          map:
            common_cbar: true
            x_pos_stats_avg: -0.1
            fontsize: 10
  plot_profiles:
    description: Plot lwcre zonal mean including reference datasets.
    variables:
      lwcre:
        timerange: 1996/2014
        mip: Amon
        preprocessor: zonal_mean
        derive: true
      swcre:
        timerange: 1996/2014
        mip: Amon
        preprocessor: zonal_mean
        derive: true
    additional_datasets:
    - dataset: CERES-EBAF
      project: OBS
      type: sat
      version: Ed4.2
      tier: 2
      start_year: 2000
      end_year: 2023
      reference_for_monitor_diags: true
    - dataset: ESACCI-CLOUD
      project: OBS
      type: sat
      version: AVHRR-AMPM-fv3.0
      tier: 2
      start_year: 1996
      end_year: 2016
    - dataset: ISCCP-FH
      project: OBS
      type: sat
      version: v0
      tier: 2
      start_year: 1996
      end_year: 2016
    scripts:
      plot:
        script: monitor/multi_datasets.py
        plot_folder: '{plot_dir}'
        plot_filename: '{plot_type}_{real_name}_{dataset}_{mip}'
        plots:
          variable_vs_lat:
            legend_kwargs:
              loc: upper right
            plot_kwargs:
              CERES-EBAF:
                color: black
              ESACCI-CLOUD:
                color: black
                linestyle: dashed
              ISCCP-FH:
                color: black
                linestyle: dotted

[bouweandela]

#2982 should solve this. The error message could be improved too. I'll work on that later this week.