Problem
cluv submit first currently does all of the syncing before any of the submitting:
# cluv/cli/submit.py: submit_first
remotes = await sync(sync_datasets=sync_datasets) # <-- barrier across all clusters
...
sbatch_results = await asyncio.gather(*[sbatch(...) for cluster, remote in cluster_to_remote.items()])
sync() fans out over the clusters with run_async_tasks_with_progress_bar, which only returns once
every cluster is done. So the await on the first line is an unnecessary synchronization barrier: no
job is submitted anywhere until the slowest cluster has finished its uv sync (plus
git fetch/checkout, fetch_results, and the dataset rsync).
Concretely: if the sync with rorqual finishes in 20s but fir takes 10 minutes (slow uv sync, cold
filesystem, ...), the rorqual job sits unsubmitted for ~10 minutes. submit first is supposed to be a
race for the earliest start time, and this hands away the head start of every fast cluster.
Expected behaviour
Each cluster should sync and then immediately sbatch, independently of the others. The race should be
across the whole per-cluster pipeline (sync → submit → wait for the job to start), with no barrier
between the sync phase and the submit phase:
- rorqual finishes syncing → its job is queued right away, and we start watching it with
sacct.
- fir is still syncing → it joins the race whenever it is ready.
- Once one job is
RUNNING, the other jobs are cancelled and the clusters that are still syncing give
up before submitting anything.
Only the genuinely shared steps need to happen up front: resolving which clusters we have a connection
to, git push, and pulling the datasets from the source cluster.
Problem
cluv submit firstcurrently does all of the syncing before any of the submitting:sync()fans out over the clusters withrun_async_tasks_with_progress_bar, which only returns onceevery cluster is done. So the
awaiton the first line is an unnecessary synchronization barrier: nojob is submitted anywhere until the slowest cluster has finished its
uv sync(plusgit fetch/checkout,fetch_results, and the dataset rsync).Concretely: if the sync with rorqual finishes in 20s but fir takes 10 minutes (slow
uv sync, coldfilesystem, ...), the rorqual job sits unsubmitted for ~10 minutes.
submit firstis supposed to be arace for the earliest start time, and this hands away the head start of every fast cluster.
Expected behaviour
Each cluster should sync and then immediately
sbatch, independently of the others. The race should beacross the whole per-cluster pipeline (sync → submit → wait for the job to start), with no barrier
between the sync phase and the submit phase:
sacct.RUNNING, the other jobs are cancelled and the clusters that are still syncing giveup before submitting anything.
Only the genuinely shared steps need to happen up front: resolving which clusters we have a connection
to,
git push, and pulling the datasets from the source cluster.