Skip to content

feat: enable retire dataset function - #31

Merged
tombrooks248 merged 1 commit into
mainfrom
feat/enable-retire-dataset-function
Oct 7, 2026
Merged

tombrooks248 merged 1 commit into
mainfrom
feat/enable-retire-dataset-function

Conversation

@tombrooks248

@tombrooks248 tombrooks248 commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

What type of PR is this? (check all applicable)

  • Refactor
  • Feature
  • Bug Fix
  • Optimization
  • Documentation Update

Description

Adds a retire mode to the Postgres loader image: task/pgload/retire.py and task/retire.sh count a dataset's rows in entity, old_entity and entity_subdivided and, unless it is a dry run, delete them in one transaction. The Dockerfile's ENTRYPOINT becomes CMD so a caller can run ./retire.sh instead of ./load.sh by passing a command, as the collection task already does. No current caller passes a command (the three Airflow DAGs only set environment variables and the task has no S3 triggers), so loading behaves exactly as before.

Why

Part 2 of digital-land/digital-land-builder#64. Changing a dataset's environment value in the specification stops it being built but leaves its entities in Postgres, so its entity pages and API results stay live. The retire-dataset DAG in airflow-dags will run this mode as its postgres task, keeping the code that removes rows next to the code that loads them. Only DRY_RUN=false deletes anything, matching the collection task's retire mode. Nothing runs this mode until the airflow-dags change is merged.

Related Tickets & Documents

QA Instructions, Screenshots, Recordings

Built the image from this branch locally and ran it against a throwaway PostGIS database:

  • With no command (how the three Airflow loader DAGs call it) the image still runs load.sh: docker inspect shows Entrypoint=null, Cmd=["./load.sh"].
  • ./retire.sh with DATASET_NAME=company and DRY_RUN unset, or mistyped as flase, counted the rows (2 entity, 1 old_entity, 0 entity_subdivided) and changed nothing.
  • DRY_RUN=false removed only the company rows; the other datasets' rows were untouched. Running it again removed 0 rows and exited 0.

Once merged, dev and staging deploy automatically. Before approving production, run manual-postgres-loader in dev on a small dataset to confirm loading still works.

Added/updated tests?

We encourage you to keep the code coverage percentage at 80% and above. Please refer to the Digital Land Testing Guidance for more information.

  • Yes
  • No, and this is why: please replace this line with details on why tests
    have not been included
  • I need help with writing tests

[optional] Are there any post deployment tasks we need to perform?

[optional] Are there any dependencies on other PRs or Work?

@tombrooks248
tombrooks248 force-pushed the feat/enable-retire-dataset-function branch from 569c08b to d7e1524 Compare October 7, 2026 13:03
@tombrooks248
tombrooks248 force-pushed the feat/enable-retire-dataset-function branch from d7e1524 to adfec94 Compare October 7, 2026 13:09
@tombrooks248
tombrooks248 merged commit 12ef8e7 into main Oct 7, 2026
4 checks passed
@tombrooks248
tombrooks248 deleted the feat/enable-retire-dataset-function branch October 7, 2026 13:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant