Skip to content

Katie b5/implement uv dependency cooldown #3075

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
KatieB5 wants to merge 2 commits into
base: KatieB5/switch-to-uv-for-updating-dependencies
Choose a base branch
from
Open

Katie b5/implement uv dependency cooldown #3075

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8bbf99d
Implement dependency cooldown using `uv`
KatieB5 May 20, 2026
d777a36
Update documentation to reflect switch to uv
KatieB5 May 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions DEVELOPERS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,51 @@ require them or build codelists. Refer to the team manual [OpenCodelists
playbook](https://bennett.wiki/tech-group/playbooks/opencodelists/) for how to
do those tasks on our infrastructure.

## Dependency management
We use [Dependabot](https://github.com/opensafely-core/opencodelists/blob/main/.github/dependabot.yml) and an [automated update workflow](https://github.com/opensafely-core/opencodelists/blob/main/.github/workflows/update-python-dependencies.yml) to keep our dependencies up to date.

Dependencies are managed with `uv`.

### Overview

See the [uv documentation](https://docs.astral.sh/uv/concepts/projects/dependencies) for details on usage.
Commands for adding, removing or modifying constraints of dependencies will automatically respect the
global timestamp cutoff specified in the `pyproject.toml`:
```toml
[tool.uv]
exclude-newer = "YYYY-MM-DDTHH:MM:SSZ"
```
Changes to dependencies should be made via `uv` commands, or by modifying `pyproject.toml` directly, followed by
[locking and syncing](https://docs.astral.sh/uv/concepts/projects/sync/) via `uv` or `just` commands like
`just devenv` or `just update-dependencies`. You should not modify `uv.lock` manually.

Note that `uv.lock` must be reproducible from `pyproject.toml`. Otherwise, `just check` and `just check-lockfile` will fail.
If either of these recipes fails with an error saying that the timestamps must match, you might have modified one file but not the other:
- If you modified `pyproject.toml`, you must update `uv.lock` via `uv lock` / `just update-dependencies` or similar.
- If you did not modify `pyproject.toml` but have changes in the `uv.lock` file, you should revert the changes to `uv.lock`, modify `pyproject.toml` as you require, then run `uv lock` to update the `uv.lock` file.

The timestamp cutoff should usually be set to midnight UTC of a past date.
In general, the date is expected to be between 7 and 14 days ago as a result of automated weekly dependency updates.

If you require a package version that is newer than the cutoff allows, you can either manually bump the global cutoff date or add a package-specific timestamp cutoff. Both options are described below.

### Manually bumping the cutoff date
The cutoff timestamp can be modified to a more recent date either manually in the `pyproject.toml` or with `just bump-uv-cutoff <days-ago>`.
For example, to set the cutoff to today's date and upgrade all dependencies, run:
```
just bump-uv-cutoff 0
just update-dependencies
```

### Adding a package-specific timestamp cutoff
It is possible to specify a package-specific timestamp cutoff in addition to the global cutoff.
This should be done in the `pyproject.toml` to ensure reproducible installs; see the [uv documentation](https://docs.astral.sh/uv/reference/settings/#exclude-newer-package) for details.
If set, the package-specific cutoff will take precedence over the global cutoff regardless of which one is more recent.

You should not set a package-specific cutoff that is older than the global cutoff - use a version constraint instead.
If there is good reason to set a package-specific cutoff that is more recent than the global cutoff, **care should be taken to ensure that the package-specific cutoff is manually removed once it is over 7 days old**, as otherwise future automated updates of that package will be indefinitely blocked.
Currently no automated tooling is in place to enforce removal of stale package-specific cutoffs.

## Local development

### Prerequisites:
Expand Down
50 changes: 47 additions & 3 deletions justfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,13 +94,57 @@ upgrade-package package: && devenv
uv lock --upgrade-package {{ package }}


# Upgrade all dev and prod dependencies to the latest versions per the pyproject.toml
# Move the cutoff date in pyproject.toml to N days ago (default: 7) at midnight UTC
bump-uv-cutoff days="7":
#!/usr/bin/env -S uvx --with tomlkit python3.13
# Note we specify the python version here and we don't care if it's different to
# the .python-version; we need 3.11+ for the datetime code used.

import datetime
import tomlkit

with open("pyproject.toml", "rb") as f:
content = tomlkit.load(f)

new_datetime = (
datetime.datetime.now(datetime.UTC) - datetime.timedelta(days=int("{{ days }}"))
).replace(hour=0, minute=0, second=0, microsecond=0)
new_timestamp = new_datetime.strftime("%Y-%m-%dT%H:%M:%SZ")
if existing_timestamp := content["tool"]["uv"].get("exclude-newer"):
if new_datetime < datetime.datetime.fromisoformat(existing_timestamp):
print(
f"Existing cutoff {existing_timestamp} is more recent than {new_timestamp}, not updating."
)
exit(0)
content["tool"]["uv"]["exclude-newer"] = new_timestamp

with open("pyproject.toml", "w") as f:
tomlkit.dump(content, f)


# Bump the timestamp cutoff to midnight UTC 7 days ago and upgrade all
# dev and prod dependencies to the latest version per pyproject.toml,
# then update the local venv.
# This is the default input command to update-dependencies action
# https://github.com/bennettoxford/update-dependencies-action
update-dependencies: && devenv
update-dependencies: bump-uv-cutoff && devenv
uv lock --upgrade


# validate uv.lock
check-lockfile:
#!/usr/bin/env bash
set -euo pipefail
# Make sure dates in pyproject.toml and uv.lock are in sync
unset UV_EXCLUDE_NEWER
rc=0
uv lock --check || rc=$?
if test "$rc" != "0" ; then
echo "Timestamp cutoffs in uv.lock must match those in pyproject.toml. See DEVELOPERS.md for details and hints." >&2
exit $rc
fi


# *ARGS is variadic, 0 or more. This allows us to do `just test -k match`, for example.
# Run the python tests, excluding the functional tests. Run coverage.
test-py *ARGS: devenv
Expand Down Expand Up @@ -131,7 +175,7 @@ test-functional *ARGS: devenv
test: assets-test test-py test-functional

# lint and check formatting but don't modify anything
check *args: devenv
check *args: check-lockfile devenv
$BIN/ruff format --diff --quiet .
$BIN/ruff check --output-format=full .
$BIN/djhtml --tabwidth 2 --check templates/
Expand Down
82 changes: 43 additions & 39 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,37 +7,37 @@ authors = [{name = "OpenSAFELY", email = "tech@opensafely.org"}]
license = {file = "LICENSE"}
requires-python = ">=3.12"
dependencies = [
"antlr4-python3-runtime>=4.13.2",
"beautifulsoup4>=4.14.3",
"crispy-bootstrap4>=2026.2",
"dj-database-url>=3.1.2",
"antlr4-python3-runtime",
"beautifulsoup4",
"crispy-bootstrap4",
"dj-database-url",
# Pin to 5.2.x until 6.0 is released and we are happy with 6.0 compatibility
"django~=5.2.13",
"django-anymail[mailgun]>=15.0",
"django-cors-headers>=4.9.0",
"django-crispy-forms>=2.6",
"django-extensions>=4.1",
"django-structlog>=10.0.0",
"django-taggit>=6.1.0",
"django-vite>=3.1.0",
"djangorestframework>=3.17.1",
"faker>=40.15.0",
"furl>=2.1.4",
"gunicorn>=25.3.0",
"lxml>=6.1.0",
"markdown2>=2.5.5",
"nh3>=0.3.5",
"openpyxl>=3.1.5",
"opentelemetry-exporter-otlp>=1.41.1",
"opentelemetry-instrumentation-django>=0.62b1",
"opentelemetry-sdk>=1.41.1",
"django>=5.2,<6.0",
"django-anymail[mailgun]",
"django-cors-headers",
"django-crispy-forms",
"django-extensions",
"django-structlog",
"django-taggit",
"django-vite",
"djangorestframework",
"faker",
"furl",
"gunicorn",
"lxml",
"markdown2",
"nh3",
"openpyxl",
"opentelemetry-exporter-otlp",
"opentelemetry-instrumentation-django",
"opentelemetry-sdk",
# pinned to mimmum version for security patch (https://github.com/opensafely-core/opencodelists/commit/99374a6aee474e6873f41f0fb2af766b9a8da61d)
"requests>=2.32.4",
"sentry-sdk>=2.58.0",
"slippers>=0.7.0",
"sqlean-py>=3.50.4.5",
"structlog>=25.5.0",
"tqdm>=4.67.3",
"sentry-sdk",
"slippers",
"sqlean-py",
"structlog",
"tqdm",
# Serves static files. Dependabot wasn't updating past 6.3.0 for
# unclear reasons, and we want fix #612 from 6.8.0.
"whitenoise[brotli]>=6.8.0",
Expand Down Expand Up @@ -125,22 +125,26 @@ lines-after-imports = 2
"codelists/views/__init__.py" = ["F401"]
"opencodelists/views/__init__.py" = ["F401"]

# Note: any `exclude-newer-package` timestamps should be removed if > 7 days old.
# See https://github.com/opensafely-core/opencodelists/blob/main/DEVELOPERS.md for details.
[tool.uv]
required-version = ">=0.9"
exclude-newer = "2026-05-18T00:00:00Z"
exclude-newer-package = {}

[dependency-groups]
dev = [
"django-debug-toolbar>=6.3.0",
"djhtml>=3.0.11",
"hypothesis>=6.152.3",
"django-debug-toolbar",
"djhtml",
"hypothesis",
# for scripts/import_codelists_from_xlsx.py
"pandas>=3.0.2",
"pandas",
# For scripts/fetch_vmp_prev.py
"paramiko>=4.0.0",
"pre-commit>=4.6.0",
"pytest-cov>=7.1.0",
"pytest-django>=4.12.0",
"pytest-playwright>=0.7.2",
"responses>=0.26.0",
"ruff>=0.15.12",
"paramiko",
"pre-commit",
"pytest-cov",
"pytest-django",
"pytest-playwright",
"responses",
"ruff",
]
Loading