Skip to content

Data Source Scrape Mode

Jump to bottom
Jakub Raczek edited this page Oct 11, 2026 · 5 revisions

Data Source: Scrape Mode

Scrape mode uses the same internal endpoints Strava's own web dashboard uses, authenticated with a browser session cookie (_strava4_session). No Strava API subscription or OAuth app required.

Two distinct scripts have scrape mode:

  • My Activities scrape mode (STRAVA_MY_SOURCE=scrape) — replaces the Strava /athlete/activities API
  • Club leaderboard scrape mode (STRAVA_SOURCE=scrape) — replaces the /clubs/{id}/activities API (deprecated 2026-09-01)

My Activities scrape mode

When to use

When your Strava API subscription lapses or you never had one.

Setup

1. Get the session cookie:

  1. Log in to strava.com in any browser.
  2. Open DevTools → Application → Cookies → www.strava.com.
  3. Find _strava4_session and copy its Value (a long alphanumeric string).

Sessions last approximately 30 days. When the script says STRAVA_SESSION_COOKIE has expired, repeat this step.

2. Edit the config:

vi /etc/strava-my-activities.conf

Add or change:

STRAVA_MY_SOURCE="scrape"
STRAVA_SESSION_COOKIE="<paste _strava4_session value here>"

The OAuth credentials (STRAVA_CLIENT_ID etc.) can remain but are ignored in scrape mode.

3. Run once to verify:

strava-my-activities

What scrape mode does

  • Uses Strava's internal /athlete/training_activities endpoint (the same one Strava's own dashboard uses)
  • Normalizes locale-formatted distances, string times, and locale dates automatically
  • Backfills per-activity detail by fetching each activity's HTML page and extracting stats (distance, time, elevation, cadence, power, gear, steps, calories)
  • Downloads the GPX export (/activities/{id}/export_gpx) and saves it alongside the detail file — the interactive route map on activity.html renders the full GPS track via Leaflet
  • Walk / Run / Hike detail — per-km splits are computed client-side from the GPX track (haversine + timestamps + elevation + HR + cadence); actual device step count is shown when the page carries it, with a cadence-based estimate as fallback

Non-destructive switch

Existing activities are matched by Strava ID and updated in place — no duplicates when switching from API mode to scrape mode.

Cookie health banner

The Data completeness page (/strava/me/data-quality.html) shows a green / amber / red status card indicating when the _strava4_session cookie was last verified and when it needs to be refreshed (~30 days). The section is hidden in API mode and when no scrape-mode config is active.

An inline form on the same page lets you paste a fresh _strava4_session value directly from the browser (DevTools → Application → Cookies → strava.com); saving it calls /cgi-bin/update-cookie, which writes the new value to both My Activities and leaderboard configs and clears the cached session files so the next sync re-verifies it.

Health alert emails

Three silent failure conditions send a consolidated alert email once per day (rate-limited by $STATE_DIR/scrape-alert.txt):

Trigger What happened
Strava layout change Detail page parse produced no usable data — Strava may have changed their HTML
Cookie expired mid-run Session cookie was rejected during detail backfill
Normalization failure jq could not parse an activity list page

Alerts go to STRAVA_MY_BIKE_EMAIL (the same address used for bike-service threshold alerts). No additional config key is needed — if that address is set, both bike and scrape alerts are active.

Testing locally

Use the provided PowerShell helper (requires Podman):

powershell -ExecutionPolicy Bypass -File test\run-scrape-test.ps1 `
    -SessionCookie "<paste _strava4_session value>" `
    -MaxPages 3 -DetailMax 10 -KeepOutput

Club leaderboard scrape mode

When to use

Strava removes the /clubs/{id}/activities API on 2026-09-01. Switch to STRAVA_SOURCE=scrape when the API stops responding.

Scrape mode gives real activity dates and Strava IDs (not first-seen approximations).

Step 1 — Get a session cookie

  1. Log in to strava.com in any browser.
  2. Open DevTools → Application → Cookies → www.strava.com.
  3. Find _strava4_session and copy its Value.

Sessions last approximately 30 days. When the script says STRAVA_SESSION_COOKIE has expired, repeat this step.

Step 2 — Update the config

vi /etc/strava-leaderboard.conf

Add or change:

STRAVA_SOURCE="scrape"
STRAVA_SESSION_COOKIE="<paste _strava4_session value here>"
STRAVA_CLUB_IDS="123456,789012"

The OAuth credentials can remain but are ignored in scrape mode.

Step 3 — Avoid overlap duplicates at transition

The Strava club feed covers roughly the last 3–4 weeks of activity. Those same activities are already in the store from the API's last few runs (with content-hash signatures). On the first scrape run, the dedup check looks for numeric IDs, not content hashes — those overlap-period activities would be added again.

To avoid this, set STRAVA_SCRAPE_START_DATE in /etc/strava-leaderboard.conf before the first scrape run:

# ~4 weeks before your switch date
STRAVA_SCRAPE_START_DATE="2026-09-01"

The script silently ignores any scraped activity whose actual start date is before this value. Once the overlap window has passed, you can comment this line out.

If you skip this, the duplication resolves within one month — duplicated activities fall out of the current-month filter and only affect all-time totals briefly. It is cosmetic, not data-destroying.

Step 4 — Deploy updated scripts

scp strava-leaderboard.sh root@192.168.1.1:/usr/bin/strava-leaderboard `
  && scp strava-lib.sh root@192.168.1.1:/usr/bin/strava-lib.sh

No install.sh re-run needed unless you changed paths or cron timing.

Healthy scrape run output

2026-09-15 23:50:01 reusing cached Strava CSRF token
2026-09-15 23:50:01 fetching club 123456 feed (cursor pagination)...
2026-09-15 23:50:02   page 1: 15 entries
2026-09-15 23:50:02   page 2: 14 entries
2026-09-15 23:50:03   page 3: empty, stopping
2026-09-15 23:50:03 fetched 29 entries from club 123456 (actual dates)
2026-09-15 23:50:03 club 123456: +5 new (actual dates), 412 total
2026-09-15 23:50:05 wrote /www/strava/activities.json and per-club leaderboard JSON (snapshot 20260915)
2026-09-15 23:50:05 wrote /www/strava/index.html
2026-09-15 23:50:05 done.

How API history and scrape history coexist

The .ndjson store is additive. API-mode entries use a content-hash string as their signature; scrape-mode entries use the numeric Strava activity ID. They never collide:

# API entry — content-hash signature, approximate first-seen date:
{"signature":"jakub|r|morning ride|34300|5400|ride","firstSeen":"2026-06-15",...}

# Scrape entry — Strava activity ID signature, real date:
{"signature":12345678901,"firstSeen":"2026-08-22",...}

Old entries keep their first-seen dates; new scrape-mode entries get real dates going forward.

How stats are parsed, and what gets dropped

The club feed returns each activity's three headline stats (stat_one, stat_two, stat_three) in an order that depends on the activity type, and labels each one with a *_subtitle (Distance, Elev Gain, Time, Pace, Cal, Avg HR, Avg Power, …). For example:

# Outdoor ride:      Distance / Elev Gain / Time
# Run:               Distance / Pace      / Time
# Indoor ride (HR):  Time     / Cal       / Avg HR   ← no Distance stat at all

The parser reads distance/elevation/time from the stat whose subtitle names it, not from a fixed position. (Older feeds without subtitles fall back to the positional assumption.) This prevents a value like "14m 6s" (Time) from being mis-read as 146 km of distance.

On top of that, four guards drop junk, so a distance-less or mis-parsed activity never lands on the distance leaderboard:

Guard Drops
distance > 0 distance-less activities (indoor cycling, yoga, weights, …)
distance ≤ 2000 km impossible single-activity distances
moving_time > 0 a non-zero distance paired with zero time — the mis-parse where an indoor ride's 1h 40m time stat is read as 140 km (its own time stat then parses to 0)
average speed ≤ 80 km/h physically-impossible speeds that flag a unit/stat mis-parse

Distances are stored as whole metres (the parser rounds km × 1000), so the .ndjson store stays free of floating-point noise.

Retroactive clean-up: these guards run in two places. They filter incoming feed entries on the way into the store, and a sanity-purge pass re-applies them to every row already in activities_<clubid>.ndjson on each run. So a bad row written by an older parser — before this activity type was understood — is removed automatically on the next strava-leaderboard run; no manual editing is needed. The run log reports it as club <id>: purged N implausible stored entries (zero-time/over-speed/over-distance). (A distance-less indoor row with distance 0 is kept — it contributes nothing to the distance board and is not a mis-parse.)

Testing locally before the cutover

Test against real Strava data without touching the router's live leaderboard. Prerequisites: Podman (or Docker), PowerShell.

# From the repo root:
$env:STRAVA_SESSION_COOKIE = "<paste _strava4_session value>"
$env:STRAVA_CLUB_IDS       = "123456,789012"

# Run scrape + open dashboard in browser:
powershell -ExecutionPolicy Bypass -File test\run-scrape-test.ps1 -Serve

# Or just run without opening browser:
powershell -ExecutionPolicy Bypass -File test\run-scrape-test.ps1

The -Serve flag starts a Python HTTP server on port 8088 (change with -Port 9090). The browser opens automatically. Press Enter to stop.

The test builds a minimal Alpine image, runs strava-leaderboard.sh with STRAVA_SOURCE=scrape, writes HTML/JSON to a Windows temp directory, and serves that directory. The router's live leaderboard is completely unaffected.

Clone this wiki locally