You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
|`LocalDataSource`| JSONL in `public/data/`|`datasets.js` entry with `source: "local"`|
22
-
|`HFDataSource`| HF Datasets Server REST API |`source: "hf"`, or `?dataset=user/name` URL param |
23
-
|`HFDiskDataSource`| Arrow IPC shards via browser file picker | "Select local dataset" button |
21
+
|`HFDataSource`| HF Datasets Server REST API |`source: "hf"`, or `?dataset=user/name` URL param, or custom user/name typed in the welcome modal |
22
+
|`HFDiskDataSource`| Arrow IPC shards via browser file picker | "Select local dataset" button (welcome modal or header) |
24
23
25
24
Factory at [src/data/index.js](src/data/index.js) dispatches by `descriptor.source`.
26
25
27
26
## Dataset selection (three ways)
28
27
29
-
1.**Hardcoded list** — `DATASETS` in [src/datasets.js](src/datasets.js). First entry is default.
30
-
2.**URL override** — `?dataset=user/name` (optional `&config=`, `&split=`, `&label=`) prepends a synthetic descriptor and selects it. Logic in `descriptorFromURL()` at the top of [src/App.jsx](src/App.jsx).
28
+
1.**Welcome modal** — on every page load, the user picks a dataset before the app loads anything. Lists known HF datasets from `DATASETS` in [src/datasets.js](src/datasets.js), accepts a custom `user/dataset_name`, or accepts a local Arrow shard directory when self-hosting.
29
+
2.**URL override** — `?dataset=user/name` (optional `&config=`, `&split=`, `&label=`) prepends a synthetic descriptor and pre-selects it in the modal. Logic in `descriptorFromURL()` at the top of [src/App.jsx](src/App.jsx).
31
30
3.**File picker** — disabled in deployed builds via `IS_DEPLOYED`. **Brittleness**: this currently checks `import.meta.env.BASE_URL !== '/'`, which works for the github.io subpath deploy but would falsely re-enable the picker for a custom-domain build (`BASE_PATH=/`). Switch to a dedicated env var like `VITE_DEPLOY_TARGET=pages` set only in the workflow if adding a custom domain.
32
31
33
32
## The summary.json mechanism
34
33
35
34
`HFDataSource.getSummary()` fetches a pre-computed `summary.json` from `https://huggingface.co/datasets/<repo>/resolve/main/summary.json`. **Without it the app degrades**: no sky map, no class filter, no class-balanced random sampling. Falls back gracefully to global random offset.
36
35
37
-
Why pre-computed: the local sources build the summary by scanning the entire dataset. Impossible for a multi-GB remote dataset, and HF's datasets-server doesn't expose per-row sky positions.
36
+
Why pre-computed: `HFDiskDataSource` builds the summary by scanning the entire dataset. Impossible for a multi-GB remote dataset, and HF's datasets-server doesn't expose per-row sky positions.
38
37
39
38
Generate with [scripts/build_summary.py](scripts/build_summary.py), upload via `huggingface-cli upload <repo> summary.json summary.json --repo-type=dataset`. Schema matches what `HFDiskDataSource._scan` produces in-memory ([src/data/HFDiskDataSource.js](src/data/HFDiskDataSource.js)) — keep them in sync.
40
39
@@ -44,7 +43,7 @@ Size scales with row count: ~1 MB for 50k rows, ~7 MB for 1M (mostly `classIndic
44
43
45
44
The sky map is capped at ~10k points (SVG render scaling). Each class contributes `min(class_size, max(SKY_FLOOR, proportional_share))`. Floor (default 100) ensures rare classes are visible; proportional share keeps common classes from dominating; min cap handles classes smaller than the floor. Total may slightly exceed budget when many small classes exist — acceptable.
46
45
47
-
Helper `sampleSkyPointsByClass()` exported from [src/data/HFDiskDataSource.js](src/data/HFDiskDataSource.js) and reused by `LocalDataSource`. The Python script [scripts/build_summary.py](scripts/build_summary.py) implements the same algorithm so summaries built locally match what the JS sources would produce.
46
+
Helper `sampleSkyPointsByClass()`is exported from [src/data/HFDiskDataSource.js](src/data/HFDiskDataSource.js). The Python script [scripts/build_summary.py](scripts/build_summary.py) implements the same algorithm so summaries built locally match what the JS source would produce.
48
47
49
48
## Lightcurve / band registry
50
49
@@ -58,8 +57,6 @@ The HF `/rows` endpoint returns nested struct/list data already in the schema-na
58
57
-`npm run build:gh` — base path `/<package_name>/` for github.io subpath. Note: `npm_package_name` from `package.json` is `timeseries-explorer`, but the GH Actions workflow overrides this with the actual repo name via `BASE_PATH=/${{ github.event.repository.name }}/`.
59
58
-[.github/workflows/deploy.yml](.github/workflows/deploy.yml) — auto-deploys to GitHub Pages on push to `main`.
60
59
61
-
The workflow currently sets `VITE_DATA_SOURCE`, `VITE_HF_DATASET`, etc. — these are dead since dataset selection moved from env vars to `src/datasets.js` + URL params. [src/config.js](src/config.js) is also vestigial. Cleanup pending.
62
-
63
60
## Style
64
61
65
62
- No comments unless the *why* is non-obvious. Don't describe what the code does.
0 commit comments