allow private bucket access w/ cloudflare auth (#1098)

* cloudflare backups worker

* cloudflare backups worker w/ private bucket access

* clarified docs

* fix backups-worker TypeScript types

* verify Cloudflare Access JWT for private backups and add public backup notice

* change from allowlist to blocklist

* update docs

---------

Co-authored-by: Pine Nguyen <pinenguyen@berkeley.edu>

Author: jayana-cpc

apps/docs/src/core/data/README.md

@@ -9,6 +9,15 @@ At its core, Berkeleytime serves as a data aggregation platform. We work directl
 
 Understanding the data sources Berkeleytime has access to is imperative for building streamlined services.
 
+## Backups and Access
+
+Production backups may contain sensitive data:
+
+- Public backups are **redacted** and are not a comprehensive dataset.
+- Full backups require Cloudflare Access.
+
+For details, see [Runbooks](../infrastructure/runbooks.md#fetch-mongo-backups).
+
 ## API Central
 
 The EIS maintains many [RESTful](https://en.wikipedia.org/wiki/REST) APIs that consolidate data from various other sources, and provides documentation in the form of [Swagger OpenAPI v3 specifications](https://swagger.io/specification/) for each API. [API Central](https://developers.api.berkeley.edu/) serves as a portal for requesting access to individual APIs, interactive documentation, and managing API usage. Berkeleytime only has access to and utilizes the APIs necessary for servicing students.

apps/docs/src/core/infrastructure/README.md

@@ -11,3 +11,7 @@ Software infrastructure refers to the services and tools that create an underlyi
 
 > [!IMPORTANT]
 > We aim to use a **small** set of **existing** infrastructure solutions with large communities. This philosophy reduces the [cognitive load](https://thevaluable.dev/cognitive-load-theory-software-developer/) on each developer and simplifies the onboarding process, both of which are valuable for creating long-lasting software in a team where developers are typically cycled out after only ~4 years.
+
+## Backups
+
+Mongo backups are served from `https://backups.berkeleytime.com`. Download steps live in [Runbooks](./runbooks.md#fetch-mongo-backups).

apps/docs/src/core/infrastructure/runbooks.md

@@ -22,6 +22,41 @@
    k create job --from cronjob/bt-prod-datapuller-courses bt-prod-datapuller-courses-manual-01
    ```
 
+## Fetch Mongo Backups
+
+Backups are served at `https://backups.berkeleytime.com`:
+
+- Public: `GET /public/*` 
+- Private: `GET /private/*`
+
+### Public backup (no auth)
+
+Public backups are meant for local development and include only a redacted subset of the `bt` database. The public backup includes these collections:
+
+- `classes`
+- `courses`
+- `terms`
+- `sections`
+- `gradeDistributions`
+- `enrollmentHistories`
+- `enrollmenttimeframes`
+
+```sh
+curl -f -o "prod_public_backup-YYYYMMDD.gz" \
+  "https://backups.berkeleytime.com/public/daily/prod_public_backup-YYYYMMDD.gz"
+```
+
+### Private backup (Cloudflare Access)
+
+```sh
+brew install cloudflare/cloudflare/cloudflared
+cloudflared access login https://backups.berkeleytime.com
+
+cloudflared access curl \
+  "https://backups.berkeleytime.com/private/hourly/prod_backup-YYYYMMDDHH.gz" \
+  -o "prod_backup-YYYYMMDDHH.gz"
+```
+
 ## Deploying a New Environment Variable with sealed-secrets
 
 Useful when adding new environment variables to `.env`. To ensure our env variables can be deployed to GitHub without their true value being leaked, they should be encrypted before being pushed to GitHub.

apps/docs/src/getting-started/local-development.md

@@ -90,7 +90,8 @@ A seeded database is required for some pages on the frontend.
 docker compose up -d
 
 # Download the data
-curl -f -o "prod-backup.gz" "https://backups.berkeleytime.com/daily/prod_public_backup-$(TZ=America/Los_Angeles date -v -6H +%Y%m%d).gz"
+curl -f -o "prod-backup.gz" "https://backups.berkeleytime.com/public/daily/prod_public_backup-$(TZ=America/Los_Angeles date -v -6H +%Y%m%d).gz"
+printf "\033[33mNotice: Public backups are redacted and are not a comprehensive dataset. Use private backups (Cloudflare Access required) for full data.\033[0m\n"
 
 # Copy the data and restore
 docker cp ./prod-backup.gz berkeleytime-mongodb-1:/tmp/prod-backup.gz

infra/base/templates/backup-prod-mongo-public.yaml

@@ -39,13 +39,25 @@ spec:
                   kubectl exec --namespace=bt \
                     "$prod_pod" -- mongodump \
                     --db bt \
-                    --collection courses \
-                    --collection classes \
-                    --collection sections \
-                    --collection terms \
-                    --collection gradeDistributions \
-                    --collection enrollmentHistories \
-                    --collection enrollmenttimeframes \
+                    --excludeCollection users \
+                    --excludeCollection plan \
+                    --excludeCollection planTerm \
+                    --excludeCollection label \
+                    --excludeCollection majorReq \
+                    --excludeCollection rating \
+                    --excludeCollection aggregatedMetrics \
+                    --excludeCollection schedules \
+                    --excludeCollection collections \
+                    --excludeCollection pods \
+                    --excludeCollection banners \
+                    --excludeCollection bannerviewcounts \
+                    --excludeCollection classviewcounts \
+                    --excludeCollection CuratedClass \
+                    --excludeCollection clickevents \
+                    --excludeCollection routeRedirects \
+                    --excludeCollection semester-roles \
+                    --excludeCollection staff-members \
+                    --excludeCollection targetedmessages \
                     --archive=/tmp/prod_public_backup.gz \
                     --gzip
 

infra/cloudflare/backups-worker/README.md

@@ -0,0 +1,65 @@
+# backups-worker – accessing backups
+
+This Worker serves Mongo backups from R2 at `https://backups.berkeleytime.com`:
+
+- `GET /public/*` → `prod-mongo-public-backups`
+- `GET /private/*` → `prod-mongo-backups`
+
+Behavior notes:
+
+- Only `GET` and `HEAD` are allowed.
+- `/private/*` requires Cloudflare Access, and the Worker cryptographically verifies `cf-access-jwt-assertion`.
+- Legacy public paths like `/daily/*` still resolve from the public bucket for compatibility.
+
+## Required private-route auth config
+
+Set these Worker variables for JWT verification:
+
+- `CLOUDFLARE_ACCESS_TEAM_DOMAIN` (for example: `your-team.cloudflareaccess.com`)
+- `CLOUDFLARE_ACCESS_AUDIENCE` (Access app AUD tag; comma-separated if you need multiple values)
+
+If either variable is missing or the token is invalid, `/private/*` returns `403`.
+
+## 1. Install `cloudflared`
+
+```bash
+brew install cloudflare/cloudflare/cloudflared
+```
+
+## 2. Fetch a public backup
+
+Public does **not** require authentication:
+
+```bash
+curl -f -o "prod_public_backup-YYYYMMDD.gz" \
+  "https://backups.berkeleytime.com/public/daily/prod_public_backup-YYYYMMDD.gz"
+printf "\033[33mNotice: Public backups are redacted and are not a comprehensive dataset. Use private backups (Cloudflare Access required) for full data.\033[0m\n"
+```
+
+Replace `YYYYMMDD` with the date.
+
+## 3. Log in for private backups
+
+Private backups require Cloudflare Access.
+
+```bash
+cloudflared access login https://backups.berkeleytime.com
+```
+
+## 4. Fetch a private backup
+
+After logging in:
+
+```bash
+cloudflared access curl \
+  "https://backups.berkeleytime.com/private/hourly/prod_backup-YYYYMMDDHH.gz" \
+  -o "prod_backup-YYYYMMDDHH.gz"
+```
+
+For monthly persistent backups:
+
+```bash
+cloudflared access curl \
+  "https://backups.berkeleytime.com/private/persistent/prod_backup-YYYYMMDD.gz" \
+  -o "prod_backup-YYYYMMDD.gz"
+```