Skip to content

data-sources.md

Latest commit

 

History

History
166 lines (141 loc) · 7.74 KB

data-sources.md

File metadata and controls

166 lines (141 loc) · 7.74 KB
Raw

Data sources

StrainScape treats every provider as an adapter. Each one normalizes a source-specific payload into the shared Signals struct, so the scoring engine never sees HTTP. Providers are composed in internal/providers/aggregate.go and run in parallel behind a shared HTTP client.

Live fan-out for a single request typically produces 8–10 live sources plus a handful of synthetic ones, depending on the location (coastal, European, US, …).

Live, no key required

Open-Meteo — geocoding + current weather

  • Forward geocoding: https://geocoding-api.open-meteo.com/v1/search
  • Reverse lookup: same endpoint with lat/lon ranking.
  • Current conditions: https://api.open-meteo.com/v1/forecast with current=temperature_2m,apparent_temperature,relative_humidity_2m, wind_speed_10m,precipitation,weather_code&timezone=auto.

The weather provider also derives the local hour from the API's timezone response, which feeds the rush-hour and late-night flags used by the Mobility and Context factors.

Open-Meteo — Air Quality

  • Endpoint: https://air-quality-api.open-meteo.com/v1/air-quality
  • Fields: pm2_5, pm10, european_aqi, us_aqi, uv_index, nitrogen_dioxide, ozone, carbon_monoxide.
  • Feeds the Environmental Load factor (AQI + PM2.5 + UV).

Open-Meteo — Pollen (CAMS Europe)

  • Same host, domains=cams_europe.
  • Fields: alder_pollen, birch_pollen, grass_pollen, mugwort_pollen, olive_pollen, ragweed_pollen in grains/m³.
  • Europe only; outside the CAMS footprint the provider reports "unavailable" and the UI surfaces the gap.

Open-Meteo — Marine

  • Endpoint: https://marine-api.open-meteo.com/v1/marine
  • Fields: wave_height, wind_wave_period.
  • Coastal only; inland lat/lon return nulls and the provider degrades.
  • Feeds the Hazard factor (rough seas).

Open-Meteo — Flood / GloFAS

  • Endpoint: https://flood-api.open-meteo.com/v1/flood
  • Fields: daily.river_discharge for the last 7 days.
  • The provider computes a z-score anomaly (today vs the 7-day history) and maps it to 0–100 to feed the Hazard factor.

USGS Earthquake Catalog

  • Endpoint: https://earthquake.usgs.gov/fdsnws/event/1/query
  • Filter: last 24 h, M ≥ 2.5, within 500 km of the request location.
  • Produces maximum magnitude, distance to nearest event, and an aftershock clustering score (number of events within 150 km).

EMSC — European-Mediterranean Seismological Centre

  • Endpoint: https://www.seismicportal.eu/fdsnws/event/1/query
  • FDSN event JSON, complementary to USGS. Faster for European and Mediterranean events.
  • The aggregator takes the stronger of the USGS and EMSC results and sums the clustering scores (capped at 100).

NOAA SWPC — planetary K index

  • Endpoint: https://services.swpc.noaa.gov/products/noaa-planetary-k-index.json
  • Global Kp 0–9; ≥5 corresponds to a geomagnetic storm.
  • Feeds the Hazard factor as a small but real "exotic" contribution.

NOAA NWS — active severe-weather alerts (US only)

  • Endpoint: https://api.weather.gov/alerts/active?point={lat},{lon}
  • Requires a descriptive User-Agent header.
  • Parses properties.severity (Minor/Moderate/Severe/Extreme) into the 0–4 ordinal used by the weather provider. Tsunami warnings are lifted to level 4.

Geopolitical country risk tier (curated)

  • Source: hand-maintained table in internal/providers/geopolitical.go.
  • Methodology: ISO-3166-1 alpha-2 country codes mapped to a 0–100 risk score. Values are set against publicly available travel- advisory and conflict-monitoring sources — primarily the UK FCDO and US State Department advisories plus the ACLED conflict dashboards — and reviewed on each release.
  • Tiers: 95 for active large-scale warfare (mass displacement, daily strikes), 85 for active armed conflict with frequent incidents, 70 for intense regional tension or high proximity to active conflict, 55 for elevated unrest, 40 for moderate elevated risk, 20 for a stable baseline, 10 for very low risk.
  • Feeds two factors: Hazard Load (geopolitical is the largest single weighted item, with a floor at geo × 0.8 for scores ≥ 80) and Information Pressure (with a floor at geo × 0.85 for scores ≥ 55). The dynamic-weighting layer also boosts hazard and information when geo ≥ 55, so conflict zones shift the composite toward those families and away from everyday signals.
  • Status: reported as synthetic in the providers array, because it's a local table rather than a real-time feed. A live adapter (ACLED with an academic key, or a travel-advisory REST endpoint) can drop in behind the same interface; the table then becomes a fallback.

MeteoAlarm — EUMETNET severe-weather alerts (40+ European countries)

  • Endpoint: https://feeds.meteoalarm.org/feeds/meteoalarm-legacy-atom-{country-slug}
  • Atom XML feed; we map each entry's embedded CAP awareness level (green/yellow/orange/red) to the same 0–4 ordinal used by NWS.
  • MeteoAlarm rejects specific XML Accept headers with 406; the provider sends Accept: */* which is the only value it honours.
  • A hand-maintained table maps ISO country codes to feed slugs. EUMETNET members only — Türkiye, Russia, and non-European countries return "unavailable" without hitting the network.

OpenSky Network — airspace activity

  • Endpoint: https://opensky-network.org/api/states/all
  • Anonymous access returns live aircraft states for a bounding box.
  • Counts active (not on-ground) aircraft in a ±0.4° box around the location and folds the result into Mobility Friction as a max() with the traffic signal — so a busy sky can lift a quiet road signal without suppressing a real road jam.
  • Cached aggressively via the 2-minute snapshot TTL to stay under the anonymous 400-credits/day ceiling.

Opt-in with an environment flag

GDELT 2.0 — news tone + volume

  • Endpoint: https://api.gdeltproject.org/api/v2/doc/doc with mode=TimelineTone.
  • Returns a tone series; average tone is mapped so -10 (very negative) → 100 and +10 (very positive) → 0, then used as the NewsNegativityScore. Timeline length becomes a volume proxy.
  • Rate-limited to ~1 request per 5 seconds per IP. The provider serializes outbound calls behind a per-instance throttle and is disabled unless GDELT_ENABLED=true is set so bursty demo traffic doesn't spam the endpoint.

Synthetic with real-provider slot

The following three providers ship with deterministic synthetic mode so the whole system works out of the box. Set the corresponding API key and swap in the real adapter — the interface is already in place.

Traffic

  • Real: TomTom Traffic API (TOMTOM_API_KEY) or HERE Flow.
  • Synthetic: per-city baseline from a city-name hash (20–74) plus a megacity boost for a small hand-maintained list of known heavy- congestion cities, plus a classical twin-peak rush-hour curve centred at 08:30 and 18:00 local time.

Economic / FX

  • Real: ECB Data Portal, national central bank feeds, or a paid exchange-rate API (EXCHANGERATE_API_KEY).
  • Synthetic: a hand-curated per-country baseline table reflecting rough real-world volatility tiers, plus a weekday drift so repeat requests feel alive.

News (fallback)

  • Real: the GDELT provider above, when enabled, replaces this feed.
  • Synthetic: per-city baseline + an hour-of-day wobble + a deterministic day-of-year "event spike" so ~15 % of days feel heavier than average.

Partial failure

Every provider returns a ProviderStatus alongside its partial Signals. The aggregator records the status for every adapter, and the HTTP response includes a providers field showing each source as live, synthetic, or unavailable. A failed provider is not a failed request: the neutral defaults in providers.DefaultSignals keep the engine running.