fix(screenscraper): stop persisting SS media download URLs (#3612)#3616
Draft
Spinnich wants to merge 1 commit into
Draft
fix(screenscraper): stop persisting SS media download URLs (#3612)#3616Spinnich wants to merge 1 commit into
Spinnich wants to merge 1 commit into
Conversation
) ScreenScraper media URLs were saved to roms.ss_metadata (the *_url entries) and to url_cover/url_manual/url_screenshots during a scan, even though the assets are downloaded to local files that the UI renders from. The stored URLs are SS API queries carrying auth params and are never read after the download, so they are an avoidable privacy/cleanliness liability. - Drop ss_metadata.*_url entirely (pop_ss_media_urls at the persist boundary). - Blank SS-sourced url_cover/url_manual/url_screenshots; other providers' public URLs are kept as valid cover fallbacks (is_screenscraper_url gate). - Persist the resolved media variant tag (e.g. "box-2D(wor)") in ss_metadata (cover_media/manual_media/screenshot_media) so the re-scan change-check still detects region/media config changes without storing a URL. - Download from the fresh in-memory URLs; credentials are re-added only at download time and never stored. - Migration 0092 cleans existing rows; frontend types regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
c91d517 to
f14abd6
Compare
Member
|
after a quick glance this seems both too specific to ssfr and too complicated for what it's trying to accomplish. i've set it as draft and will revisit after 5.0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
ScreenScraper media URLs were saved into the database during a scan, even though every asset is also downloaded to a local file that the UI actually renders from. They lived in two places on the
romstable:url_cover/url_manual/url_screenshotscolumns, and*_urlentries inside thess_metadataJSON blob.These URLs are ScreenScraper API queries that carry authentication parameters (
devid/devpassword/ssid/sspassword) and are never read after the in-scan download. Persisting them writes authentication detail into the DB (and into/api/roms/*responses) for no functional gain — an avoidable risk for anyone who backs up or shares their database, logs, or support bundles. This closes #3612.The approach: the SS URLs are now held only in memory long enough to drive the download, then dropped before anything is persisted. Because the assets are keyed by the stored
ss_id(non-sensitive) and every scan that downloads SS art already re-queries ScreenScraper, the URLs are always re-derivable and never need to be stored.The one thing the stored URL was still used for after a scan is the re-scan "has this asset changed?" check. To preserve that without storing a query, we now persist a tiny, non-sensitive variant tag instead — the resolved
mediadescriptor, e.g.box-2D(wor)— inss_metadata(cover_media/manual_media/screenshot_media). On a re-scan, comparing that tag detects when a config change (e.g. a new region priority) selects a different asset, so UPDATE-scan refresh keeps working. Other metadata providers' URLs (IGDB, Moby, …) are plain public CDN links and are left untouched.What changed and why
ss_metadata.*_urlis dropped at the persist boundary; SS-sourcedurl_cover/url_manual/url_screenshotsare blanked. Non-SS provider URLs are kept as valid cover fallbacks.screenscraper.fr, and never store them anywhere.Files modified
backend/handler/metadata/ss_handler.py— Core logic. Strip the full SS auth param set (ssid/sspassword/devid/devpassword) from in-memory media URLs; re-add the full set inadd_ss_auth_to_urlat download time. New helpers:pop_ss_media_urls(drop the*_urlkeys),_ss_media_descriptor(extract themediatag), publicis_screenscraper_url(host gate, renamed from a private helper). NewSSMetadatatag fields and tag computation inbuild_ss_game.backend/endpoints/sockets/scan.py— Capture the SS media URLs and blank the SS-sourced url columns beforeadd_rom; download cover/manual/screenshots and special media from the captured URLs; change the cover/manual/screenshotsoverwritedecision to "stored URL changed (non-SS) OR variant tag changed (SS)."backend/endpoints/roms/__init__.py— Manual-edit /ss_id-change flow: drop*_urland blank SS-sourcedurl_cover/url_manualbeforeupdate_rom(after the download has consumed them).backend/alembic/versions/0092_strip_ss_media_urls.py— Batched data migration: drop every*_urlfromss_metadata, and blankurl_cover/url_manual/ drop SS entries fromurl_screenshotsonly when they point at ScreenScraper. Cross-dialect (typedCustomJSON). Downgrade is intentionally a no-op (the stripped data can't be reconstructed; local assets remain valid).backend/tests/handler/metadata/test_ss_handler.py— Tests for auth stripping, dev-cred re-attachment,pop_ss_media_urls,is_screenscraper_url,_ss_media_descriptor, andbuild_ss_gametag output.backend/tests/alembic/test_0092_strip_ss_media_urls.py(+__init__.py) — Unit tests for the migration's transform helpers (host gating, URL blanking,*_urlremoval).frontend/src/__generated__/models/RomSSMetadata.ts— Regenerated types;RomSSMetadatagained the three optional tag fields (additive, backward-compatible).Testing notes
trunk fmt && trunk check(ruff/black/isort/mypy/bandit) clean.upgrade head→downgrade -1→upgrade head.url_coverstored blank,cover_mediatag stored,ss_metadata.*_urlabsent.200); covers stayed put as expected, since RomM's region selection is dominated by each rom's own filename region tag.Reviewer attention
overwritedecision inscan.pyis the core new logic (url changed OR SS variant tag changed). Most relevant on UPDATE scans.store_media_fileskips if the file exists), so it refreshes only on a Complete rescan while cover/screenshots refresh on UPDATE. This asymmetry is pre-existing and intentionally left out of scope; happy to file a follow-up.path_*).ss_metadatano longer contains*_urlvalues and gains the three optional tag fields; existing consumers read*_path, not*_url.Checklist
AI assistance disclosure: This PR — the code, the migration, the tests, and this description — was written primarily by Claude Code (Opus), under my direction and review. I verified the behavior against real scans locally as described above.