Journal Trends — backend

Flask + SQLite cache and SEO-prerender backend for journaltrends.com. Companion to the frontend repo at Journal-Trends/journal-trends.

A project of India Research Watch. MIT licensed.

---

What it serves

Path	Purpose
GET /api/stats/<issn>	Cached aggregation for one journal (Scopus + PPS joined). 404 if not cached.
POST /api/stats/<issn>	Frontend writes a new aggregation after a fresh OpenAlex scrape. Verified against OpenAlex + per-IP rate-limited before insert.
GET /api/stats	List of all cached ISSNs (debug).
GET /api/pps/version	Current PPS dataset version + count.
GET /journal/<slug>-<issn>	SEO-prerendered HTML — journal-named <title>, OG cards, JSON-LD, full SPA hydrate.
GET /journal/<slug>-<issn>.png	Per-journal 1200×630 OG thumbnail rendered with PIL.
GET /sitemap.xml	Dynamic sitemap of all cached journals.
GET /healthz	Liveness probe.

The Netlify frontend proxies /journal/* and /sitemap.xml to this backend transparently so the URLs stay on journaltrends.com.

Data sources

• OpenAlex — publication metadata. Validated on every POST.
• Problematic Paper Screener — flagged-paper integrity signals (loaded from a slim gzipped CSV via scripts/load_pps.py).
• Scopus Source List (Elsevier ext_list.xlsx, monthly) — Active/Inactive indexing status + delisting history (loaded via scripts/build_scopus.py then scripts/load_scopus.py).

Quick start

python -m venv .venv
source .venv/bin/activate            # Windows: .venv\Scripts\Activate.ps1
pip install -r requirements.txt
python app.py                         # serves on http://localhost:5000

In dev mode, when the sibling journal-trends/ frontend repo is present in the parent directory, Flask also serves its static assets — so one python app.py runs the full stack at localhost:5000.

Deploy on PythonAnywhere (free tier works)

# In a PA Bash console
cd ~
unzip ~/journal-trends-backend-vX.Y.Z.zip -d journal-trends-backend
cd journal-trends-backend
pip3 install --user -r requirements.txt

In PA's Web tab:

• Manual config, Python 3.10+
• Source code: /home/<user>/journal-trends-backend
• WSGI file: from app import app as application after adding the source dir to sys.path

Then touch /var/www/<user>_pythonanywhere_com_wsgi.py to reload. Hit https://<user>.pythonanywhere.com/healthz — should return ok.

cache.db (SQLite) is created on first request, next to app.py. The bundled data/scopus_*.csv.gz files are loaded on first run of load_scopus.py.

Loading / refreshing data

Problematic Paper Screener (occasional refresh)

# Locally — slim the 190 MB PPS export to ~10 MB (filters Status='Genuine')
python scripts/slim_pps.py /path/to/all_problematic_papers_YYYY-MM-DD.csv
# Upload the resulting data/*.slim.csv.gz to PA
# Then on PA:
python3 scripts/load_pps.py

Scopus extension list (monthly)

Elsevier publishes an updated ext_list_<Mon>_<Year>.xlsx each month.

# Locally — strip the xlsx down to two gzipped CSVs
python scripts/build_scopus.py /path/to/ext_list_<Mon>_<Year>.xlsx
# Upload data/scopus_*.csv.gz + data/scopus_meta.json to PA
# Then on PA:
python3 scripts/load_scopus.py

Journal-name autocomplete index (monthly)

Walks OpenAlex /sources and writes the bundled autocomplete index to the FRONTEND repo:

python scripts/build_journal_index.py
# Then commit ../journal-trends/data/journal_index.json

Scripts overview

Script	Purpose
scripts/slim_pps.py	Slim + gzip the PPS export; filters out Status='Genuine' (re-reviewed legitimate papers).
scripts/load_pps.py	Load slim PPS gzip into the pps_papers table.
scripts/build_scopus.py	Strip the monthly Elsevier ext_list.xlsx into two CSVs + a meta.json.
scripts/load_scopus.py	Atomically replace the scopus_sources + scopus_discontinued tables; bulletproof against partial-write states.
scripts/build_journal_index.py	Walks OpenAlex /sources at works_count > 100 and writes the bundled name→ISSN index to the frontend repo.
scripts/sync_frontend_template.py	Copy frontend index.html into templates/journal.html so the SEO prerender stays in sync.
scripts/seed_delisted.py	Walk the Scopus discontinued list and cache every delisted journal's aggregation locally — pulls from the deployed PA cache first, falls back to OpenAlex. Dumps long-format CSVs for offline analysis.
scripts/backfill_aggregations.py	Retrofit the 'Unknown' country bucket + 'Other' institution bucket onto already-cached aggregations, no re-scrape needed. Idempotent; dry-run by default.

See CHANGELOG.md for the per-release history.

Schema versioning

SCHEMA_VERSION in app.py must match the frontend aggregator's SCHEMA_VERSION. Bumping it invalidates the cache without deleting it — rows with older versions become 404s on GET and are overwritten on the next POST.

License

MIT — see LICENSE.