Replace the pytorch example with a distributed ImageNet training example

[lebrice]

Replaces the CIFAR-10 pytorch-example stub with a port of the mila-docs ImageNet
example, adapted
to run on any configured cluster using one job script per cluster (job_script_path).

The old example never trained anything and never used more than one GPU (it had a TODO: Make this a distributed example, so that it can also run on Tamia and others), so it didn't exercise the parts of
cluv that matter for real work: multi-node jobs, per-cluster resource differences, dataset staging into
$SLURM_TMPDIR, checkpoint/requeue.

What the example does

examples/pytorch-example/ → examples/imagenet/, with the Mila-specific parts replaced by cluv's
runtime API:

• checkpoints and wandb files go to current_run_info().results_path instead of a hand-rolled
  $SCRATCH/checkpoints symlink
• prepare_data.py reads the ILSVRC2012 archives from the cluster's resolved datasets_path instead
  of a hardcoded /network/datasets/imagenet
• upstream's safe_sbatch / code_checkpointing.sh are dropped: cluv submit already enforces a
  clean tree and exports $GIT_COMMIT

Each cluster gets its own scripts/job_<cluster>.sh holding nothing but #SBATCH directives, which
execs a shared scripts/train.sh. --use_fake_data also skips the dataset staging step, so a cheap
smoke job can run on a cluster that has no copy of ImageNet.

Runs on the clusters

Every row is a real job submitted with cluv submit from this branch, with no job script named on
the command line - the per-cluster job_script_path picks it. Runtimes are sacct Elapsed, which
includes the per-node venv warm-up and (for real-data rows) the ImageNet extraction.

Cluster	GPUs requested	Job ID	Runtime	Dataset	W&B
mila	2× L40S	10286668	13m24s	real ImageNet	mila_10286668
mila	2× L40S	10286635	38 s	fake	--no_wandb
tamia	4× H100 (whole node)	395938	42 s	fake	offline
fir	4× H100	53007584	7m03s	fake	offline
rorqual	4× H100	18361699	55 s	fake	offline
narval	4× A100-40GB	147008	queued	fake	offline
nibi	4 of 8× H100	19105826	queued	fake	offline
killarney	4× L40S	4605084	running	fake	offline
vulcan	4× L40S	345697	queued	fake	offline
trillium-gpu	4× H100 (whole node)	715096	queued	fake	offline

Every completed run trained, validated, wrote epoch_0.pt/epoch_1.pt, and produced one profiler
trace per rank. The real-ImageNet run on mila spent 11m37s of its 13m24s extracting the archives into
$SLURM_TMPDIR, then trained at ~2500 images/second on 2 L40S.

The remaining rows are queued on the DRAC/Vector schedulers; the grid will be updated with their
runtimes. A cluv sync is separately replicating the ~150GB of ImageNet archives from mila to every
cluster (the local pull is done; the pushes take hours), after which real-data runs can follow
elsewhere.

Bugs this shook out

Getting one example to run on nine clusters turned up seven real defects. Each is a separate commit;
the cluv-side ones carry regression tests.

In cluv:

1. current_run_info() returned None for the first task of every job — not SLURM_PROCID
   treats 0 as unset, so rank 0 disagreed with the other ranks about the run id and results path.
2. --no-sync-datasets didn't skip the push. sync_datasets was only consulted for the
   data_source pull; sync_task_function decided whether to push from config.data_source alone,
   so the flag silently rsynced the whole dataset to every cluster anyway.
3. A per-cluster project_dir couldn't relocate a subproject. clone_project derived the repo
   path from $HOME whenever the local repo was under $HOME, ignoring project_dir. Killarney
   refuses jobs submitted from /home, and Trillium doesn't mount /home on compute nodes, so
   neither could work at all.
4. cluv submit had no --no-sync-datasets, although cluv sync does — so you couldn't submit
   a job while a cluv sync was replicating a dataset.
5. cluv submit now exports $CLUV_CLUSTER, and current_cluster() prefers it. A cluster
   doesn't always call itself by the name used to reach it: jobs submitted to trillium-gpu report
   CC_CLUSTER=trillium and Slurm's ClusterName there is grillium, so the job resolved the wrong
   [tool.cluv.clusters.<name>] section. Killarney and Vulcan only set CC_CLUSTER in a login shell.

In the example / job scripts:

6. $SLURM_SUBMIT_DIR is the wrong variable for locating a helper script — it is where sbatch
   was invoked ($HOME over SSH), not cluv's --chdir target, so every job died with
   bash: /home/<user>/scripts/train.sh: No such file or directory.
7. --gpus-per-task=1 breaks DDP — cgroups expose one GPU per task, so
   torch.cuda.set_device(LOCAL_RANK) fails with invalid device ordinal in every rank but the
   first. --gres-flags=allow-task-sharing (which upstream relies on) had no effect on tamia, so the
   scripts allocate GPUs with --gpus-per-node.

Plus a performance trap worth knowing about: on Lustre-backed $HOME, four ranks faulting in the
same ~2GB of torch shared libraries at once left fir stuck in cl_sync_io_wait for over five
minutes. scripts/train.sh now reads the venv once per node first.

Known cluv issues this PR only works around

Two problems that are cluv-wide, not example-specific. Both are worked around in this example's
config rather than fixed in cluv, because the fixes reach much further than this PR.

1. Slurm on the Vector clusters (Killarney, Vulcan) does not create the parent directory of
--output. cluv's default is {results_path}/{cluster}_%j/slurm-%j.out, so the job is killed
before it starts — FAILED, exit 0:53, no output file at all. Verified by A/B test: the identical
job with a flat output path runs fine.

2. $SCRATCH in a cluv-computed path can expand to nothing. cluv submit runs
bash --login -c '<env vars> sbatch ... <args>', but the shlex-quoted arguments are concatenated
into that single-quoted string and close it, so the paths are expanded by the non-login shell ssh
starts — not by the login shell. On most clusters $SCRATCH is set in both and nothing looks wrong;
on Killarney and Vulcan it is login-shell-only, and Slurm recorded
StdOut=/logs/imagenet/<jobid>.out. Anything else relying on $SCRATCH in a cluv-computed path
fails the same way there. The real fix is to quote the inner command once
(bash --login -c {shlex.quote(inner)}), which changes every submission cluv makes and deserves its
own PR.

Both are dodged here with an output in sbatch_args that is relative to the job's working
directory, resolved through the logs symlink cluv already creates in the project folder.

Dataset handling

data_source = "mila:/network/datasets/imagenet" with datasets_path = "$SCRATCH/datasets/imagenet"
uniformly on every cluster. datasets_path is deliberately not pointed at the shared read-only
copies that some clusters have (/network/datasets on mila, the rrg-bengioy-ad curated copy on
fir, /datashare/imagenet on nibi), because cluv sync rsyncs into that path — aiming it at a
shared dataset folder would try to write there. The README lists the shared copies for anyone who
would rather read them directly and skip the sync.

One caveat worth knowing: since the data_source cluster is also a sync target, cluv sync pushes a
redundant copy back to mila's own $SCRATCH. Jobs on mila don't use it — current_cluster_config()
rewrites datasets_path to the data_source path when running on the source cluster.

Tests

• New test_imagenet_example (slow, per-cluster) submits the fake-data smoke job with no job script
  argument, so per-cluster job_script_path resolution is what's under test.
• New fast test_example_job_scripts_exist asserts every configured job_script_path exists locally —
  cluv reads the script header before submitting, so a typo would otherwise only surface at submit time.
• uv run pytest -m "not integration": 162 passed. mkdocs build --strict clean.

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

❌ Patch coverage is 73.68421% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.80%. Comparing base (aa2ba35) to head (2d2741c).

Files with missing lines	Patch %	Lines
cluv/cli/sync.py	66.66%	4 Missing ⚠️
cluv/cli/submit.py	66.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #164      +/-   ##
==========================================
+ Coverage   63.78%   65.80%   +2.02%     
==========================================
  Files          19       19              
  Lines        2079     2091      +12     
==========================================
+ Hits         1326     1376      +50     
+ Misses        753      715      -38     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.