Merge branch 'main' into cu-h2o-rdf

# Conflicts:
#	uv.lock

Author: IsaacParker30

.gitignore

@@ -40,6 +40,7 @@ pip-wheel-metadata/
 __pycache__/
 ml_peg/app/data/*
 !ml_peg/app/data/onboarding/
+!ml_peg/app/data/table_download/
 certs/
 ml_peg/calcs/surfaces/copper_water_interface/data/
 ml_peg/calcs/aqueous_solutions/bulk_water/data/

.jules/bolt.md

@@ -0,0 +1,7 @@
+## 2024-05-19 - Caching YAML Load for Model Parsing
+**Learning:** `yaml.safe_load` on large configuration files like `models.yml` is significantly slower than parsing JSON or doing other basic IO. It can become a bottleneck when called repeatedly throughout an application's lifecycle (e.g., getting subsets of models, instantiating apps).
+**Action:** Always memoize or `@lru_cache` functions that load static, read-only configuration files (like `models.yml`) to prevent repeated disk I/O and parsing overhead.
+
+## 2024-05-19 - Caching YAML Load for Framework Registry
+**Learning:** `yaml.safe_load` on `frameworks.yml` within `load_framework_registry()` was taking ~2-3 ms per call and it was repeatedly called for every framework entry via `get_framework_config()`. This was a micro-bottleneck, especially when dealing with lists or multiple frameworks.
+**Action:** Applied the `@lru_cache` and `deepcopy` pattern successfully again to `load_framework_registry()` and `get_framework_config()` to avoid caching a mutable dictionary directly and avoid repeated YAML I/O parsing.

.pre-commit-config.yaml

@@ -2,7 +2,7 @@
 # pre-commit install
 repos:
 - repo: https://github.com/pre-commit/pre-commit-hooks
-  rev: v5.0.0
+  rev: v6.0.0
   hooks:
     - id: end-of-file-fixer
     - id: mixed-line-ending
@@ -11,7 +11,7 @@ repos:
 
 - repo: https://github.com/astral-sh/ruff-pre-commit
   # Ruff version.
-  rev: v0.11.13
+  rev: v0.15.5
   hooks:
     # Run the linter.
     - id: ruff-check
@@ -20,7 +20,7 @@ repos:
     - id: ruff-format
 
 - repo: https://github.com/numpy/numpydoc
-  rev: v1.8.0
+  rev: v1.10.0
   hooks:
     - id: numpydoc-validation
       files: ^ml_peg/

README.md

@@ -40,7 +40,7 @@ python3 -m pip install git+https://github.com/ddmms/ml-peg.git
 
 ## Features
 
-Coming soon!
+More details coming soon!
 
 
 ## Development
@@ -66,6 +66,53 @@ Please refer to the [online documentation](https://ddmms.github.io/ml-peg/develo
 for information about contributing new benchmarks and models.
 
 
+### Command-line interface
+
+To help run calculations, analysis, and the application, we provide the `ml_peg`
+command line tool, which is installed with the package. This provides the following
+commands:
+
+```shell
+ml_peg app
+ml_peg calc
+ml_peg analyse
+ml_peg download
+ml_peg list
+```
+
+For example, to run the X23 test with mace-mp-0a and orb-v3-consv-inf-omat, you can run:
+
+```shell
+ml_peg calc --test X23 --models mace-mp-0a,orb-v3-consv-inf-omat
+```
+
+A description of each subcommand, as well as valid options, can be listed using the
+`--help` option. For example:
+
+```shell
+ml_peg calc --help
+```
+
+The `ml_peg list` command provides a further set of subcommands:
+
+```shell
+ml_peg list calcs
+ml_peg list analysis
+ml_peg list app
+ml_peg list models
+```
+
+which list the available tests and categories that may be run for `ml_peg calc`,
+`ml_peg analyse` and `ml_peg app`, and the MLIPs that these can be run for.
+
+
+### Tutorials
+
+We encourage developers new to the ML-PEG framework to work through the detailed
+step-by-step guides provided by our Jupyter Notebook tutorials:
+- [Adding a New Benchmark](docs/source/tutorials/python/adding_benchmark.ipynb) [![badge](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ddmms/ml-peg/blob/main/docs/source/tutorials/python/adding_benchmark.ipynb)
+
+
 ## Docker/Podman images
 
 You can use [Docker](https://www.docker.com) or [Podman](https://podman.io/) to build

conftest.py

@@ -8,7 +8,7 @@
 
 import pytest
 
-from ml_peg.models import models
+from ml_peg import models
 
 
 def pytest_addoption(parser):
@@ -31,6 +31,12 @@ def pytest_addoption(parser):
         default=None,
         help="MLIPs, in comma-separated list. Default is all models",
     )
+    parser.addoption(
+        "--models-file",
+        action="store",
+        default=None,
+        help="Filepath to model definitions. Default models.yml in models directory.",
+    )
 
 
 def pytest_configure(config):
@@ -41,6 +47,9 @@ def pytest_configure(config):
 
     # Set current models from CLI input
     models.current_models = config.getoption("--models")
+    model_file = config.getoption("--models-file")
+    if model_file:
+        models.models_file = model_file
 
 
 def pytest_collection_modifyitems(config, items):

docs/source/developer_guide/add_benchmarks.rst

@@ -34,6 +34,16 @@ the error with respect to DFT predictions of energies.
 Reference data may also include experimental data, or higher-accuracy theoretical predictions,
 e.g. CCSD(T).
 
+Each metric in ``metrics.yml`` should have a ``level_of_theory`` field identifying its reference
+method. The app compares this string against the ``level_of_theory`` set in ``models.yml`` for each
+MLIP, flagging mismatches using the traffic light system. An exact string match is required.
+
+.. warning::
+
+    Use the standard strings defined in :doc:`Levels of theory </developer_guide/levels_of_theory>`.
+    A typo or inconsistent capitalisation (e.g. ``experiment`` instead of ``Experimental``) will
+    cause incorrect warnings.
+
 In some cases, metrics may also encode correct behaviour without a specific reference,
 such as by quantifying features of a known distribution (curvature, minima, etc.),
 or quantifying the stability of a simulation.
@@ -99,7 +109,7 @@ the same calculation is run for each model name-model pair:
 .. code-block:: python3
 
     from ml_peg.models.get_models import load_models
-    from ml_peg.models.models import current_models
+    from ml_peg.models import current_models
 
     MODELS = load_models(current_models)
     DATA_PATH = Path(__file__).parent / "data"
@@ -340,7 +350,7 @@ CLI (``ml_peg analyse``), a subset can be used using the ``--models`` option.
     from ml_peg.app import APP_ROOT
     from ml_peg.calcs import CALCS_ROOT
     from ml_peg.models.get_models import get_model_names
-    from ml_peg.models.models import current_models
+    from ml_peg.models import current_models
 
     MODELS = get_model_names(current_models)
     CALC_PATH = CALCS_ROOT / [category] / [benchmark_name] / "outputs"
@@ -501,3 +511,72 @@ Building those components and their interactivity should become increasingly aut
 but less standard plots/interactions will need setting up.
 
 For now, please contact us to help with this process.
+
+Framework credit and custom group tags
+++++++++++++++++++++++++++++++++++++++
+
+If a benchmark comes from an external benchmarking framework (for example, MLIP Audit),
+or you want to group benchmarks by a custom category (such as those involved in a
+paper), you can add a framework credit tag as follows:
+
+1. Add/update the framework entry in ``ml_peg/app/utils/frameworks.yml``.
+
+.. code-block:: yaml
+
+    mlip_audit:
+    label: MLIP Audit
+    color: "#1d4ed8"
+    text_color: "#ffffff"
+    url: "https://huggingface.co/spaces/InstaDeepAI/mlipaudit-leaderboard"
+    logo: "https://raw.githubusercontent.com/instadeepai/mlipaudit/mlpeg-migration/InstaDeep_Logo.png"
+
+2. Set ``framework_ids`` in the benchmark app constructor. This accepts either a
+   single framework identifier or a sequence of them, so a benchmark can belong to
+   multiple frameworks at once.
+
+   The app constructor also provides a ``include_ml_peg`` parameter. This is ``True``
+   by default, which automatically adds the built-in ``ml_peg`` tag to benchmarks, so
+   ``framework_ids`` only needs to list *additional* or *alternative* frameworks. If
+   you want to omit this tag, set ``include_ml_peg=False``.
+
+   For example:
+
+.. code-block:: python3
+
+    # MLIP Audit only
+    return SomeBenchmarkApp(
+        name="SomeBenchmark",
+        ...,
+        framework_ids="mlip_audit",
+        include_ml_peg=False,
+    )
+
+
+    # MLIP Audit and the MACE Multihead paper
+    return SomeBenchmarkApp(
+        name="SomeBenchmark",
+        ...,
+        framework_ids=["mlip_audit", "mace-multihead"],
+        include_ml_peg=False,
+    )
+
+    # ML-PEG (added by default) and the MACE Multihead paper
+    return SomeBenchmarkApp(
+        name="SomeBenchmark",
+        ...,
+        framework_ids="mace-multihead",
+    )
+
+That is all that is required. The benchmark header shows one badge per framework,
+and the additional framework pages for non-default frameworks are populated
+automatically from this metadata.
+
+Framework sections group matching benchmarks by category, omit the category
+summary table, and reuse the same benchmark tables and controls. Updating
+weights or thresholds there therefore updates the same benchmark views shown in
+the category pages.
+
+The ``logo`` field is optional. It can point to a remote image URL or a local
+Dash asset path such as ``/assets/frameworks/my_framework_logo.png``. Use a
+browser-supported image format such as ``.svg``, ``.png``, or ``.jpg``/``.jpeg``;
+``.pdf`` is not supported. For best results, use a square logo image.