Skip to content

testrun: TESTRUN_LAUNCHER for batch systems; tests/reconverse-site-run.sh for HPC-site runs of the reconverse tier - #3975

Merged
ritvikrao merged 1 commit into
reviewed-with-reconversefrom
site-run
Sep 12, 2026
Merged

testrun: TESTRUN_LAUNCHER for batch systems; tests/reconverse-site-run.sh for HPC-site runs of the reconverse tier#3975
ritvikrao merged 1 commit into
reviewed-with-reconversefrom
site-run

Conversation

@lvkale

@lvkale lvkale commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Runtime-core changes get a manual run on an HPC site before merging (the reconverse-ci header says so), and until now that meant a hand-written sbatch script each time; the Anvil validation of reconverse #217 and #222 wrote one. This makes it a one-line job that tracks CI.

  • testrun honors TESTRUN_LAUNCHER: a launcher plus flags, e.g. srun --mpi=pmi2 -N2 --ntasks-per-node=1 -c2, used for every run including single-process ones, with -n <procs> appended. Inside a Slurm allocation a reconverse binary must go through srun even alone (LCI's bootstrap reads the inherited SLURM_NTASKS/SLURM_PROCID and waits for peers). Unset, nothing changes: bare exec, or lcrun for TESTRUN_PROCS > 1.
  • tests/reconverse-site-run.sh runs the reconverse-ci.yaml TEST_DIRS tier inside an allocation: build each directory, then make test once single-process and once as PROCS processes over NODES nodes, one RESULT <dir> <shape> exit=<code> line per run and a summary. SITE=anvil loads that machine's modules and hwloc library path; SITE=delta sets FI_PROVIDER=cxi; launcher and flags are overridable (which also lets it run with lcrun on a workstation).

Usage, from an allocation such as salloc -N 2 -n 4 -c 2 or an sbatch script, after building charm:

SITE=anvil CHARM_BUILD=$PWD/reconverse-linux-x86_64 tests/reconverse-site-run.sh
SITE=delta NODES=2 CHARM_BUILD=$PWD/reconverse-linux-x86_64 tests/reconverse-site-run.sh

Checked on macOS with lcrun standing in for srun (LAUNCHER=.../lcrun LAUNCHER_ARGS_SINGLE= LAUNCHER_ARGS_MULTI= NODES=1): the three testrun modes produce the expected commands and the script runs a two-directory tier end to end. Not yet exercised on Anvil or Delta itself; the first site run of it is the remaining test.

🤖 Generated with Claude Code

@lvkale @claude
testrun: TESTRUN_LAUNCHER for batch systems; tests/reconverse-site-ru…
757f66f 
…n.sh

Runtime-core changes get a manual run on an HPC site before merging, and
until now that meant a hand-written sbatch script per run (the Anvil
validation of reconverse #217/#222 wrote one). Two pieces make it a
one-line job that tracks CI:

- testrun honors TESTRUN_LAUNCHER, a launcher plus its flags ("srun
  --mpi=pmi2 -N2 --ntasks-per-node=1 -c2"), used for every run including
  single-process ones, with -n <procs> appended. Inside a Slurm allocation a
  reconverse binary must go through srun even alone: LCI's bootstrap reads
  the inherited SLURM_NTASKS/SLURM_PROCID and waits for peers. Unset, the
  behaviour is unchanged (bare exec, or lcrun for TESTRUN_PROCS > 1).

- tests/reconverse-site-run.sh runs the reconverse-ci TEST_DIRS tier (read
  from the workflow file, so the site run and CI stay in step) inside an
  allocation: build each directory, then make test once as a single process
  and once as PROCS processes over NODES nodes, printing one RESULT line
  per run. SITE=anvil loads that machine's modules and hwloc path;
  SITE=delta sets the cxi provider; the launcher and its flags are
  overridable, which also makes the script runnable with lcrun on a
  workstation (used to check it here).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@ritvikrao
ritvikrao merged commit 8d21fda into reviewed-with-reconverse Sep 12, 2026
4 checks passed
@lvkale

lvkale commented Sep 13, 2026

Copy link
Copy Markdown
Contributor Author

Post-merge run of tests/reconverse-site-run.sh on Anvil (charm 8d21fda, submodule 58921e9; gcc 11.2, libfabric 1.12, hwloc 1.11.13; built with --with-cmake-args="-DRECONVERSE_ENABLE_CPU_AFFINITY=ON -DHWLOC_ROOT_DIR=$RCAC_HWLOC_ROOT"). Ritvik ran it on Delta before the merge.

job shape result
highmem, 1 node, script defaults 1proc + 2proc-1node, 13 dirs 16/26 pass; the five +p6 zerocopy dirs (zerocopy_with_qd, large_bcast, bcast_nonzero_root, dereg_and_nodereg, zc_post_async) exit=2 in both shapes
same, LAUNCHER_ARGS_SINGLE='-N1 -c8' LAUNCHER_ARGS_MULTI='-N1 -c4' same 26/26 pass
wholenode, 2 nodes, defaults 1proc + 2proc-2nodes same five dirs fail
same, LAUNCHER_ARGS_SINGLE='-N1 -c8' LAUNCHER_ARGS_MULTI='-N2 --ntasks-per-node=1 -c8' same 26/26 pass, multi-process runs span both nodes

One finding about the script. The default launcher flags (-N1 -c4 single, -c2 per task multi) give each srun step a cpuset smaller than the PE count of the tier's +p6 cases, and reconverse's affinity check then aborts with "Multiple PEs assigned to same core", which reads like a zerocopy regression but is allocation sizing. The header's suggested salloc -N 2 -n 4 -c 2 is below the minimum too. Fix options: defaults of -c8/-c4, or derive -c from the tier's largest +p. Cosmetic: RESULT ... exit=2 carries make's exit code, not the abort reason, and megatest's skipped +p3 multi-process case prints as testrun(reconverse): +pe 3 not divisible by 2 processes. Skipping the test. and counts as a pass.

Everything else worked as documented: SITE=anvil module loads and LD_LIBRARY_PATH, TESTRUN_LAUNCHER routing single-process runs through srun --mpi=pmi2 -n 1 (no PMI hang), +p to +pe translation. Raw logs on Anvil under x-lkale/software/charm3975/results/.

@lvkale lvkale mentioned this pull request Sep 13, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ritvikrao ritvikrao ritvikrao approved these changes
@adityapb adityapb Awaiting requested review from adityapb adityapb is a code owner
@ericjbohm ericjbohm Awaiting requested review from ericjbohm ericjbohm is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@lvkale @ritvikrao