Skip to content

cluv submit first waits for every cluster to finish syncing before submitting any job #165

Description

@lebrice

Problem

cluv submit first currently does all of the syncing before any of the submitting:

# cluv/cli/submit.py: submit_first
remotes = await sync(sync_datasets=sync_datasets)   # <-- barrier across all clusters
...
sbatch_results = await asyncio.gather(*[sbatch(...) for cluster, remote in cluster_to_remote.items()])

sync() fans out over the clusters with run_async_tasks_with_progress_bar, which only returns once
every cluster is done. So the await on the first line is an unnecessary synchronization barrier: no
job is submitted anywhere until the slowest cluster has finished its uv sync (plus
git fetch/checkout, fetch_results, and the dataset rsync).

Concretely: if the sync with rorqual finishes in 20s but fir takes 10 minutes (slow uv sync, cold
filesystem, ...), the rorqual job sits unsubmitted for ~10 minutes. submit first is supposed to be a
race for the earliest start time, and this hands away the head start of every fast cluster.

Expected behaviour

Each cluster should sync and then immediately sbatch, independently of the others. The race should be
across the whole per-cluster pipeline (sync → submit → wait for the job to start), with no barrier
between the sync phase and the submit phase:

  • rorqual finishes syncing → its job is queued right away, and we start watching it with sacct.
  • fir is still syncing → it joins the race whenever it is ready.
  • Once one job is RUNNING, the other jobs are cancelled and the clusters that are still syncing give
    up before submitting anything.

Only the genuinely shared steps need to happen up front: resolving which clusters we have a connection
to, git push, and pulling the datasets from the source cluster.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions