reconverse-site-run: SITE=frontier, larger default -c, abort reason on RESULT lines - #3977
Merged
Merged
Conversation
…t reasons
Frontier (OLCF) case: FI_PROVIDER=cxi and LCI_NETWORK_BACKENDS=ofi as on
delta, PMI_MAX_KVS_ENTRIES=1000 for Cray PMI's key-value store, and
--network=single_node_vni on every srun step. The flag goes on the
multi-node argument set too: a Makefile's +p1 cases run as one process,
Slurm shrinks that step to one node, and without the flag the cxi
provider aborts in LCI ("Function not implemented"). It is harmless on
steps that do span nodes. Module loads stay with the caller, as for
delta.
Raise the launcher defaults, which were too small for the tier's +p6
cases and made reconverse abort with "Multiple PEs assigned to same
core": LAUNCHER_ARGS_SINGLE -N1 -c4 -> -N1 -c8, LAUNCHER_ARGS_MULTI
-c2 -> -c4 on one node and -c8 per task across nodes; the header's
salloc example now says -c 8.
When a run fails, append the runtime's "Reason:" line (or the C++
what() text when the network layer threw) to its RESULT line, so a
launcher or cpuset abort is not read as a test failure.
Verified on Frontier 2026-09-13, reviewed-with-reconverse 8d21fda with
reconverse 58921e9: all 13 CI directories pass as 1 process and as
2 processes on one node (job 5481253) and on two nodes (job 5481302).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
adityapb
approved these changes
Sep 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Script-only follow-up to #3975 (
tests/reconverse-site-run.sh, +34/-8, no runtime code). Written and validated on Frontier by the on-machine session, 2026-09-13.Changes
frontier)case:FI_PROVIDER=cxi,LCI_NETWORK_BACKENDS=ofi,PMI_MAX_KVS_ENTRIES=1000(Cray PMI's default of 30 entries is too few for a multi-process launch), and--network=single_node_vnion both launcher argument sets. The last one is the finding: the Makefiles'+p1cases run as one process, Slurm shrinks that step to one node, and a one-node step on Frontier has no VNI without the flag, so LCI's cxi provider aborts with "Function not implemented" (and leaves an 870 MB core file). A probe job confirmed the flag is harmless on steps that span nodes. Module loads stay the caller's job, as in thedeltacase.LAUNCHER_ARGS_SINGLE-N1 -c4->-N1 -c8;LAUNCHER_ARGS_MULTI-c2->-c8per task multi-node and-N1 -c4one node. The tier's+p6cases need a cpuset of at least 6 per process or reconverse aborts with "Multiple PEs assigned to same core" (the Anvil run of 2026-09-13 on the merged testrun: TESTRUN_LAUNCHER for batch systems; tests/reconverse-site-run.sh for HPC-site runs of the reconverse tier #3975 hit this in five directories). Header's salloc example fixed to match.RESULT ... exit=Nnow appends the run'sReason:line (CmiAbort) or the C++what():text (LCI exception), so a launcher or cpuset abort is not read as a test failure.Validation of the tier through this script
--with-production, libfabric 2.3.1 cxi)+p1cases on a shrunk one-node step-cvalues this PR makes the default; 16/26 with the old defaultsFrontier details are in charm-notes
machines/frontier.md(a55e95f) and Anvil's inmachines/anvil.md(992ca10).🤖 Generated with Claude Code