Skip to content

Commit 04a755d

Browse files
committed
Refine CLAUDE.md documentation: update data source descriptions and clarify dataset selection methods
1 parent 74a8561 commit 04a755d

1 file changed

Lines changed: 7 additions & 10 deletions

File tree

CLAUDE.md

Lines changed: 7 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -14,27 +14,26 @@ Vite + React 18, no TypeScript, no router. Plotly.js (`react-plotly.js` + `plotl
1414
- `getRows({ offset, length })` → array of plain row objects
1515
- `getSummary()``{ totalRows, classCounts, classIndices, bands, skyPoints }` or `null`
1616

17-
Three implementations swap behind it:
17+
Two implementations swap behind it:
1818

1919
| Source | Origin | When it's used |
2020
|---|---|---|
21-
| `LocalDataSource` | JSONL in `public/data/` | `datasets.js` entry with `source: "local"` |
22-
| `HFDataSource` | HF Datasets Server REST API | `source: "hf"`, or `?dataset=user/name` URL param |
23-
| `HFDiskDataSource` | Arrow IPC shards via browser file picker | "Select local dataset" button |
21+
| `HFDataSource` | HF Datasets Server REST API | `source: "hf"`, or `?dataset=user/name` URL param, or custom user/name typed in the welcome modal |
22+
| `HFDiskDataSource` | Arrow IPC shards via browser file picker | "Select local dataset" button (welcome modal or header) |
2423

2524
Factory at [src/data/index.js](src/data/index.js) dispatches by `descriptor.source`.
2625

2726
## Dataset selection (three ways)
2827

29-
1. **Hardcoded list**`DATASETS` in [src/datasets.js](src/datasets.js). First entry is default.
30-
2. **URL override**`?dataset=user/name` (optional `&config=`, `&split=`, `&label=`) prepends a synthetic descriptor and selects it. Logic in `descriptorFromURL()` at the top of [src/App.jsx](src/App.jsx).
28+
1. **Welcome modal**on every page load, the user picks a dataset before the app loads anything. Lists known HF datasets from `DATASETS` in [src/datasets.js](src/datasets.js), accepts a custom `user/dataset_name`, or accepts a local Arrow shard directory when self-hosting.
29+
2. **URL override**`?dataset=user/name` (optional `&config=`, `&split=`, `&label=`) prepends a synthetic descriptor and pre-selects it in the modal. Logic in `descriptorFromURL()` at the top of [src/App.jsx](src/App.jsx).
3130
3. **File picker** — disabled in deployed builds via `IS_DEPLOYED`. **Brittleness**: this currently checks `import.meta.env.BASE_URL !== '/'`, which works for the github.io subpath deploy but would falsely re-enable the picker for a custom-domain build (`BASE_PATH=/`). Switch to a dedicated env var like `VITE_DEPLOY_TARGET=pages` set only in the workflow if adding a custom domain.
3231

3332
## The summary.json mechanism
3433

3534
`HFDataSource.getSummary()` fetches a pre-computed `summary.json` from `https://huggingface.co/datasets/<repo>/resolve/main/summary.json`. **Without it the app degrades**: no sky map, no class filter, no class-balanced random sampling. Falls back gracefully to global random offset.
3635

37-
Why pre-computed: the local sources build the summary by scanning the entire dataset. Impossible for a multi-GB remote dataset, and HF's datasets-server doesn't expose per-row sky positions.
36+
Why pre-computed: `HFDiskDataSource` builds the summary by scanning the entire dataset. Impossible for a multi-GB remote dataset, and HF's datasets-server doesn't expose per-row sky positions.
3837

3938
Generate with [scripts/build_summary.py](scripts/build_summary.py), upload via `huggingface-cli upload <repo> summary.json summary.json --repo-type=dataset`. Schema matches what `HFDiskDataSource._scan` produces in-memory ([src/data/HFDiskDataSource.js](src/data/HFDiskDataSource.js)) — keep them in sync.
4039

@@ -44,7 +43,7 @@ Size scales with row count: ~1 MB for 50k rows, ~7 MB for 1M (mostly `classIndic
4443

4544
The sky map is capped at ~10k points (SVG render scaling). Each class contributes `min(class_size, max(SKY_FLOOR, proportional_share))`. Floor (default 100) ensures rare classes are visible; proportional share keeps common classes from dominating; min cap handles classes smaller than the floor. Total may slightly exceed budget when many small classes exist — acceptable.
4645

47-
Helper `sampleSkyPointsByClass()` exported from [src/data/HFDiskDataSource.js](src/data/HFDiskDataSource.js) and reused by `LocalDataSource`. The Python script [scripts/build_summary.py](scripts/build_summary.py) implements the same algorithm so summaries built locally match what the JS sources would produce.
46+
Helper `sampleSkyPointsByClass()` is exported from [src/data/HFDiskDataSource.js](src/data/HFDiskDataSource.js). The Python script [scripts/build_summary.py](scripts/build_summary.py) implements the same algorithm so summaries built locally match what the JS source would produce.
4847

4948
## Lightcurve / band registry
5049

@@ -58,8 +57,6 @@ The HF `/rows` endpoint returns nested struct/list data already in the schema-na
5857
- `npm run build:gh` — base path `/<package_name>/` for github.io subpath. Note: `npm_package_name` from `package.json` is `timeseries-explorer`, but the GH Actions workflow overrides this with the actual repo name via `BASE_PATH=/${{ github.event.repository.name }}/`.
5958
- [.github/workflows/deploy.yml](.github/workflows/deploy.yml) — auto-deploys to GitHub Pages on push to `main`.
6059

61-
The workflow currently sets `VITE_DATA_SOURCE`, `VITE_HF_DATASET`, etc. — these are dead since dataset selection moved from env vars to `src/datasets.js` + URL params. [src/config.js](src/config.js) is also vestigial. Cleanup pending.
62-
6360
## Style
6461

6562
- No comments unless the *why* is non-obvious. Don't describe what the code does.

0 commit comments

Comments
 (0)