What happened
PR #74 (302079b) reverted ecmwf-aifs-single-forecast-virtual from production back to staging. Its commit message notes:
Note: src/upload.py only ever uploads, so the already-published stac.dynamical.org/ecmwf-aifs-single-forecast-virtual/collection.json object must be deleted from R2 by hand.
src/upload.py's upload() walks the generated stac/**/*.json tree and PUTs each file — it never deletes objects that are no longer generated (e.g. because a CatalogItem was removed or flipped to staging=True). So removing a dataset from production leaves its old collection.json object orphaned in R2 until someone manually deletes it via a separate, out-of-band step.
That manual deletion isn't coordinated with the automated catalog.json publish (upload-stac.yml, triggered on push to main) or with stac.dynamical.org's CDN cache. If the orphaned object is deleted around the same time the new catalog.json (with the link already removed) is propagating, there's a window where a client can fetch a still-cached catalog.json that lists the child link while the underlying collection.json object is already gone — a straight 404.
This is exactly what broke Sentry issue WXOPTICON-C: wxopticon's monitor for the unrelated noaa-hrrr-forecast-48-hour-virtual dataset failed for ~2 minutes (14 events, 2026-08-10 18:06:27–18:08:27 UTC, right at the PR #74 merge/deploy) because dynamical_catalog.load_catalog() fetches every child collection listed in the root catalog and fails the whole call if any single one fails — an unrelated dataset's transient unpublish 404 took down monitoring for a dataset nobody was touching.
It self-healed once the cache/deploy settled, so no data was lost, but the same race will recur on every future unstaging (or dataset removal) and can affect any consumer of dynamical_catalog.load_catalog(), not just wxopticon.
Suggested fix
Make upload() (or a wrapper around it) delete orphaned R2 objects that are no longer present in the generated stac/ tree, so publish is a single atomic-ish step instead of "upload new files, then remember to hand-delete the old one later." At minimum, diff the bucket's existing keys against the generated tree and delete anything no longer produced (scoped to collection/item JSON, not unrelated bucket contents).
If a full sync-with-delete is too risky to automate safely, a smaller fix: document/automate deleting the specific orphaned key(s) as part of the same PR/deploy that removes them from catalog.json, rather than as a manual follow-up step, so there's no window where the two are out of sync.
Separately worth considering (not blocking): dynamical_catalog.load_catalog()'s all-or-nothing fetch semantics mean one bad collection blocks every dataset's lookup. That's arguably a dynamical-catalog design question rather than a dynamical-stac bug, so I'm not filing that here, but flagging it since it's what turned this transient 404 into a cross-dataset monitoring outage.
Filed from Sentry issue triage (WXOPTICON-C).
What happened
PR #74 (302079b) reverted
ecmwf-aifs-single-forecast-virtualfrom production back to staging. Its commit message notes:src/upload.py'supload()walks the generatedstac/**/*.jsontree and PUTs each file — it never deletes objects that are no longer generated (e.g. because aCatalogItemwas removed or flipped tostaging=True). So removing a dataset from production leaves its oldcollection.jsonobject orphaned in R2 until someone manually deletes it via a separate, out-of-band step.That manual deletion isn't coordinated with the automated
catalog.jsonpublish (upload-stac.yml, triggered on push tomain) or withstac.dynamical.org's CDN cache. If the orphaned object is deleted around the same time the newcatalog.json(with the link already removed) is propagating, there's a window where a client can fetch a still-cachedcatalog.jsonthat lists the child link while the underlyingcollection.jsonobject is already gone — a straight 404.This is exactly what broke Sentry issue
WXOPTICON-C: wxopticon's monitor for the unrelatednoaa-hrrr-forecast-48-hour-virtualdataset failed for ~2 minutes (14 events, 2026-08-10 18:06:27–18:08:27 UTC, right at the PR #74 merge/deploy) becausedynamical_catalog.load_catalog()fetches every child collection listed in the root catalog and fails the whole call if any single one fails — an unrelated dataset's transient unpublish 404 took down monitoring for a dataset nobody was touching.It self-healed once the cache/deploy settled, so no data was lost, but the same race will recur on every future unstaging (or dataset removal) and can affect any consumer of
dynamical_catalog.load_catalog(), not just wxopticon.Suggested fix
Make
upload()(or a wrapper around it) delete orphaned R2 objects that are no longer present in the generatedstac/tree, so publish is a single atomic-ish step instead of "upload new files, then remember to hand-delete the old one later." At minimum, diff the bucket's existing keys against the generated tree and delete anything no longer produced (scoped to collection/item JSON, not unrelated bucket contents).If a full sync-with-delete is too risky to automate safely, a smaller fix: document/automate deleting the specific orphaned key(s) as part of the same PR/deploy that removes them from
catalog.json, rather than as a manual follow-up step, so there's no window where the two are out of sync.Separately worth considering (not blocking):
dynamical_catalog.load_catalog()'s all-or-nothing fetch semantics mean one bad collection blocks every dataset's lookup. That's arguably adynamical-catalogdesign question rather than adynamical-stacbug, so I'm not filing that here, but flagging it since it's what turned this transient 404 into a cross-dataset monitoring outage.Filed from Sentry issue triage (WXOPTICON-C).