GPU-aware load balancer - #3968
Open
adityapb wants to merge 17 commits into
Open
Conversation
Plan item 11. The hooks were merged ahead of this (HAPI_CUPTI_LB behind an ifdef nothing defined, the comm buffer's LB region, +gpulbbuffer); this makes them do something and adds the strategy that consumes them. Measurement (hapi_impl.cpp, gpumanager.h, devicemanager.h): - Per-object attribution through CUPTI external correlation, stamped around every entry method in CkCallstackPush/Pop rather than around kernel launches, so it does not matter how the application launches its kernels. The correlation ID is a process-local token for the full LB identity (GpuObjectTokenTable): CkMigratable::ckGetID() alone loses the object-manager and aliases equal element IDs from different chare arrays. - HAPI_CUPTI_NO_OBJECT is UINT64_MAX, not 0: element IDs start at 0, so a zero sentinel silently drops one object's load. - CUPTI_ACTIVITY_KIND_RUNTIME must stay enabled even though nothing reads its records -- EXTERNAL_CORRELATION records are only emitted for correlation IDs generated by runtime-API tracking, and without it every object reads zero. - Two-pass correlation join: activity buffers complete out of order, so a kernel can be parsed before the correlation naming it. Only such kernels are parked for the second pass. - Forced flush (CUPTI_ACTIVITY_FLAG_FLUSH_FORCED) before the drain, and the correlation map is cleared after it: entry-method correlation emits a record per runtime call, not per launch, and the non-kernel remainder would otherwise grow without bound. - hapiNormalizeCuptiLoads turns the kernel timeline into SM-utilization- normalized load by a per-device sweep-line, splitting each interval's device time among the kernels holding the device in proportion to the SMs each occupies. Occupancy-seconds instead would report a kernel that holds the device at low occupancy as almost no load, and the error is not uniform: a node holding many small-grid objects reads as idle and a balancer sends it still more work. - Tracing follows LBTurnInstrumentOn/Off, counted per PE so one PE closing its window does not stop recording for the others mid-iteration. Building the round's loads is gated on the LAST PE of the process arriving (hapiCuptiArrive), because the records are process-wide while the LB's barrier is per-PE. - The measurement window closes where the load is read, not at MigrationDone: otherwise kernels running through the strategy and the migrations belong to no round at all, and how many are discarded depends on how long the step took -- a feedback loop between migration count and measured load. Plumbing: LDObjData::gpuTime and gpuPupSize, ProcStats::gpu_device_id and gpu_total_sms, carried in CLBStatsMsg. CentralLB::InvokeLB builds the round's loads and continues in gpuLoadsReady. setObjGPUTime/getObjGPUTime were declared on this line but never defined; they are defined now. Strategy: GreedyRefineCentralGPULB, two-level greedy refine -- GPU load across GPU groups (the PEs sharing a device), CPU load across the PEs within a group. The data structures are guarded by CMK_CUDA, not HAPI_CUPTI_LB: they live in LDObjData, which ck-core, ck-ldb and hybridapi all see, and a per-target macro would give them different layouts. HAPI_CUPTI_LB guards only the CUPTI calls, so a CUDA build without CUPTI still compiles and reads zero GPU load. Removes the dormant event-timing scaffold it supersedes: hapiLaunchKernel was declared but never implemented, so enabling HAPI_CUPTI_LB would have exposed an undefined symbol, and HAPI_LAUNCH_KERNEL_WRAPPER timed kernels through an hev.obj path whose setObjGPUTime counterpart did not exist. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A GPU-aware balancer that cannot move a chare's device state can only balance applications that rebuild it on arrival. This adds the device half of a migration, so a chare that keeps state on the GPU can be moved like any other. PUP (pup.h, pup_util.C): PUPMode::DEVICE marks a buffer as device-resident, and `p(ptr, n, PUPMode::DEVICE)` routes it through the migration stream's device region instead of its host region. The sizer reports gpu_size() alongside size(); the memory walkers take an optional device region. Unlike pup_buffer, this does not allocate -- an unpacking chare allocates its device buffer and then pups into it -- because the destination is what decides where the state lives. The device region carries no description of its own layout: the receiver reconstructs it by making the same sequence of pup calls the sender made. That only works if both agree on where each buffer starts, which is what DEVICE_PUP_ALIGN fixes, applied identically by the sizer, the packer and the unpacker. The device bottleneck is bytes_device(), a distinct name rather than an overload of bytes(). As an overload it is hidden in every subclass that overrides the host bottleneck alone -- which is all but three of them -- and every compiler that warns about a partially overridden virtual then says so in user code that merely includes pup.h (observed as nvcc #611-D against the gpudirect examples). Its default drops the buffer, which is what every PUP::er without a device region should do: sizing for a checkpoint, writing to disk and converting to text all operate on host memory and cannot dereference a device pointer at all. The consequence is worth stating: this migrates device state, it does not checkpoint it. Transport (cklocation): the source packs its device state into a staging buffer and sends it as a nocopydevice parameter to immigrateGPU, so the existing device zerocopy layer picks the transport -- same-process copy, CUDA IPC or device RDMA -- rather than this path choosing one. Staging rather than handing over the element's own buffers is what lets the element be destroyed as soon as the pack finishes; only the staging buffer has to outlive the transfer, and the destination's ack (finishGPUSend) releases it. The two halves are independent sends and arrive in either order, so whichever lands first waits for the other. The landing buffer is recorded under the element's id in the post method rather than recovered from the delivery method's pointer argument: for a non-SDAG entry method the generated dispatch passes the posted pointer only on the zerocopy leg, and the delivery leg sees a null. It is also moved from posted to received only once the transfer has landed, or immigrate would unpack from memory the transport has not written. The payload is sent from its own entry method rather than inline in emigrate, because at that point the element being torn down is still the running one and the payload belongs to the runtime. setObjGPUTime/getObjGPUTime's siblings setGPUPupSize/CkLocRec::setGPUPupSize were likewise declared but never defined; they are defined here, and the device footprint is reported exactly rather than through pupSize's encoded approximation, which is tuned for host state. tests/charm++/cuda/gpumigrate: two device buffers per element, the second deliberately not a multiple of DEVICE_PUP_ALIGN so that a packer and unpacker disagreeing about the second buffer's offset is caught; buffers poisoned on arrival before the device pup, because cudaMalloc may hand back the source's just-freed memory with the old contents intact and the test would otherwise pass while doing nothing. Verified against a negative control (device pup commented out aborts as expected). Driven with ckMigrate() rather than a load balancer so the device path is tested on its own. Verified: 16 blocks, 5 rounds, 4 PEs on an RTX 3050, three consecutive runs. That is the same-process transport only; IPC and RDMA need more than one process and more than one node respectively, and are what a reviewer should run (see the test's README). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… migrates Any centralized balancer built with CMK_GLOBAL_LOCATION_UPDATE=1 aborts on the first migration it decides: Assertion itr->second.pe == CkMyPe() failed: cklocation.C CkLocCache::recordEmigration <- CkLocMgr::emigrate <- LBDatabase::Migrate <- CentralLB::ProcessReceiveMigration ReceiveMigration walked every move calling UpdateLocation(move) before acting on it, on every PE -- including the PE the object is leaving. That writes cache[id].pe = move.to_pe, and emigrate's own recordEmigration then asserts the entry still says CkMyPe(), because recording the departure is precisely its job: it is what turns "the element is here" into "it is at the destination", and it checks that it was here to begin with. Move the call into the branch that runs on neither end of the move. The destination does not need it either -- it learns its own location when the element lands, through createLocal -> CkLocCache::insert -- and taking it there would bump the epoch ahead of the arriving message's, which insert also asserts on. Reproduced on this line before any GPU work: 0b01bbb configured with EXTRA_OPTS="-DCMK_GLOBAL_LOCATION_UPDATE=1", jacobi2d-imbalance, 4 PEs, +balancer MetisLB. It is invisible in a default build because the option is off, and CI does not exercise it; it surfaces here because device zerocopy asks for the option (see the abort text in ckrdmadevice.C), so a GPU build is exactly where it bites. After the fix, on the same configuration: MetisLB, RecBipartLB, GreedyRefineCentralGPULB and DistributedLB all complete 30 iterations with 3 load-balancing steps. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Kale's standing rule is that user-facing docs change with the change, and this series added a balancer and a build requirement that nothing announced. The balancer entry says what the two levels are and, more usefully, what its load number means: CUPTI-measured device demand normalized to whole-device occupancy, so it estimates what an object would cost on a different GPU rather than only what it cost where it ran. That is the property that makes it usable as a placement estimate at all, and it is not evident from the name. The prerequisite is CMK_GLOBAL_LOCATION_UPDATE=1. Device zerocopy sends are addressed to the PE the sender believes hosts the target, so without the global update a send issued around a migration reaches a PE that no longer hosts it and CkRdmaDeviceIssueRgets aborts -- correctly, rather than reading the wrong buffer, but the abort text is the only place this was written down and it is reached only after the fact. Handling that arrival instead of aborting needs the device-payload correction protocol, which is deliberately not in this series. Noted in the test README too, with the reason it does not change that test's own result: gpumigrate sends no device zerocopy messages of its own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot stopped work on behalf of
adityapb due to an error
September 10, 2026 21:03
…still link CI's netlrts-linux-x86_64-cuda job failed to link every `-language converse++` program (tests/util/check, benchmarks/converse/commbench) with a wall of undefined references out of libhybridapi: _lb_args, CkActiveLocRec(), CkCallback::send, CkCallback::impl_thread_init, CkCallback::thread_destroy. The cause is one line in pup_util.C. hapiCheck expands to hapiErrorDie, which lives in hapi_impl.cpp, and pup_util.o is part of LIBCONV_UTIL -- linked into programs that have no libck. Referencing it from there makes the linker pull hapi_impl.cpp.o out of the archive, and with it every Charm++ symbol that object uses. That is also why CkCallback::send appears: it has always been in hapiPollEvents, and was simply never reached from a converse-only link before. Use cudaMemcpy directly with a CmiAbort on failure. cudart is already in CMK_LIBS for a CUDA build, so this keeps the device copies without leaving the Converse level. Also drop hapi_impl.cpp's new dependency on _lb_args. It was gating two diagnostic printfs, and reading a ck-ldb global to do it added one more way to trip the same invariant. The verbosity is now CHARM_LB_CUPTI_DEBUG, matching the CHARM_GPU_LOAD_AUDIT and CHARM_LB_CUPTI_TIME switches already in the file, and LBManager.h gives way to lbdb.h, which supplies the LB types this file actually needs and is header-only. CkActiveLocRec and CkCallback remain: hapi is reached only from ck-core/init.C, so hapi_impl.cpp.o is only ever pulled into links that have libck. That is the same position the base line is in; the comment now records it, because when the invariant is broken the failure surfaces in unrelated programs rather than here. Verified by compiling both translation units and reading their symbol tables: pup_util.o now needs only cudaMemcpy and cudaGetErrorString, and no hapi or Charm++ symbol at all; hapi_impl.o no longer names _lb_args. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…equires The merge brings in that option's documentation and an assert enforcing it: migrations must happen at load balancing steps, and CkLocMgr::multiHop now aborts if a message ever needs a second hop to find its element. Two things on this branch did not meet that contract. CentralLB: the earlier fix here excluded both ends of a move from UpdateLocation. Only the source may be excluded -- it updates its own entry through emigrate -> recordEmigration, which asserts the element was still here. The destination does eventually learn the location for itself, when the element lands in createLocal, but until then it still believes the element is on the source, which has already let it go. Every message it sends in that window takes an extra hop, which is exactly what multiHop now catches. Update on the destination as well; skip only the source. gpumigrate: the test drove migration with ckMigrate(), i.e. migrateMe(), which the manual now states is unsupported in this mode. It moves through AtSync with RotateLB instead -- the strategy documented for exercising pup routines, and the only one that will move a set of identically loaded objects. This also makes the test cover the path a GPU application actually migrates on, rather than a shortcut around the load balancer. Driving it through a balancer introduces a way for the test to pass while doing nothing: if no object moves, the buffers are never packed, sent or unpacked, and every check trivially succeeds. ResumeFromSync now fails when the element is still on the PE it started the step on. Confirmed by running without a balancer, where it aborts naming the missing +balancer RotateLB. Verified on the merged tree, 4 PEs, CMK_GLOBAL_LOCATION_UPDATE=1: jacobi2d-imbalance completes under MetisLB, RecBipartLB, GreedyRefineCentralGPULB and DistributedLB, and gpumigrate passes 16 blocks over 5 rounds. Before this commit MetisLB aborted in multiHop. The manual note for the GPU balancer now points at the new Global Location Update section instead of restating it, and carries the AtSync restriction. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…gBal GreedyRefineCentralGPULB has every PE compute one candidate assignment from a different (A,B) parameter pair, and picks a winner in receiveSolutions. Two things decide that winner. A candidate is feasible if its migration count is within +LBPercentMovesAllowed, and among feasible candidates one with fewer migrations wins only if its max load is within LOAD_MIG_BAL of the best. Both defaults pull the same way. The migration budget defaults to 100%, so every candidate is feasible, and LOAD_MIG_BAL was a hardcoded 1.003, so a candidate had to be within 0.3% of the best max load before its lower migration count counted for anything. PE 0 deliberately runs regular greedy (A=0, B=-1, migrations ignored) and therefore usually holds the lowest max load. When object loads are close to uniform, every migration-averse candidate is more than 0.3% behind it, so the greedy one wins and nearly every object moves. Measured on jacobi2d-imbalance, 256 chares on 32 PEs, 3 LB steps, one A40 node, varying only this value with the migration budget left at its default: 1.003 (old hardcoded) 513 moves 0.184 s final max/avg 1.152 1.05 275 moves 0.130 s 1.232 1.20 7 moves 0.061 s 1.173 So the default is not a tradeoff that has been tuned; it is tight enough that it rarely binds at all, and the result is a full reshuffle bought for a fraction of a percent of max load. Make it _lb_args.loadMigBal(), default unchanged at 1.003, settable with +LBLoadMigBal. This is the better-shaped of the two controls: it says how much max load you will give up to avoid migrating, rather than capping move count outright. Note a value near the step's load spread binds erratically -- 1.05 gave 3/245/27 moves across three steps -- so choose one clear of it. The #define is removed rather than kept as a default, so the constant and the reasoning about why it is tight live in one place, beside the field. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Make.cidepends maps each generated header back to the interface-file stamp that produces it, one line per module: MetisLB.decl.h MetisLB.def.h: MetisLB.ci.stamp GreedyRefineCentralGPULB had no such line, so make knew of no rule producing GreedyRefineCentralGPULB.decl.h. Everything else asks for that header: Makefile_lb.sh lists the balancer in COMMON_LDBS, so Make.lb adds it to LBHEADERS and headerlinks takes it as a prerequisite, and the generated CommonLBs.ci and EveryLB.ci both declare it as an extern module. The legacy build was told the header was required and never told how to build it, and stopped with No rule to make target 'GreedyRefineCentralGPULB.decl.h', needed by 'headerlinks'. Add the missing line, in the file's sorted position. This only ever broke the legacy Makefile build, which is why it survived: CMake does not read Make.cidepends at all, it has its own module targets in src/ck-core/CMakeLists.txt, so every CMake build passed while CI could not work. Adding a balancer needs both registrations, and only one of them is exercised locally. Verified with the CI's own command, ./build all-test netlrts-linux-x86_64 cuda: reaches "Built target all-test", no "No rule to make target" in the log, the header is generated, and both of the CI's post-build assertions pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tests/charm++/cuda/gpumigrate carried seven files that were never meant to be committed: six .btr crash backtraces and the test's own 4.9 MB executable. The backtraces are from runs on someone's laptop -- they name a hostname and paths under /home/aditya/charm-lb, and each records a process terminated with signal 6. Nothing references them and they are not fixtures. The binary is an unstripped ELF with debug info, sitting beside the sources that produce it. They exist only on this branch; main, origin/main and rate-aware-gpu-lb are all clean, so there is nothing to tidy elsewhere. Ignore both so they do not come back. *.btr goes in the root .gitignore, since a crash dump lands beside whatever binary crashed rather than only this test. The executable gets a per-directory .gitignore naming it, which is what tests/ampi/exit already does for its own executables. The extension is not written anywhere in charm's sources or in reconverse on its current branch, so the comment on the rule does not name a producer. The four real files -- Makefile, README.txt, gpumigrate.C, gpumigrate.ci -- are untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GreedyRefineCentralGPULB balances across GPU groups on LDObjData::gpuTime, which is filled in from CUPTI activity records. CUPTI is CUDA-only and nothing equivalent is wired up for HIP, so on a HIP build every object's gpuTime stays zero: the cross-group stage sums zeros and only the within-group CPU pass does any work. That is silent today -- the balancer looks like it is balancing on GPU load while its primary dimension is empty. Say so, once, from the constructor, which is where this balancer is asked for and so the precise point at which GPU load becomes a requirement. Putting it in CentralLB instead would fire for every central balancer, including the ones that never look at GPU load. It prints even under quiet mode, because it reports degraded behaviour rather than being a banner line. No gating changes were needed for the CUDA side: every CUPTI reference in the tree already sits inside #if CMK_CUDA -- the cupti.h include, the declarations in hapi.h, the LDObjData fields, the LBObj accessors and the CentralLB paths -- and CUPTI_LAUNCH_WRAPPER already has a no-op #else. Note that HAPI_CUPTI_LB is NOT a second spelling of CMK_CUDA: cmake defines it only when it finds the CUPTI library, so a CUDA build without CUPTI has one and not the other, which is what the inner gate and its stubs are for. Verified that the block compiles and emits the string by compiling a copy with the gate forced on; reconverse's generated converse_config.h defines CMK_HIP unconditionally, so -DCMK_HIP=1 on the command line does not reach it. Charm has not been built for HIP end to end here, so the message has not been seen to print in a real HIP run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Drops the environment-gated block in hapiNormalizeCuptiLoads that printed a
[gpu-audit] line with kernel counts, attributed and unowned device-seconds,
utilisation and occupancy.
Four accumulators existed only to feed that print and go with it:
sweep_attr_demand, sweep_unattr_demand and sweep_busy_s, along with their
declarations and accumulation sites, and sweep_unattr_kernels, which had no
reader left at all.
Two of those were tangled with real control flow, which is preserved. The
unattributed test read
if (!k.attributed) { sweep_unattr_demand += k.demand; continue; }
where the continue is the logic and only the accumulation was diagnostic; it is
now a plain continue. The normalisation itself, cupti_obj_norm_load_ += demand,
is untouched, so measured load is unchanged.
The comment above cuptiDebugLevel listed this flag beside CHARM_LB_CUPTI_TIME
as the file's environment-driven diagnostics; it now names only the one that
remains. CHARM_LB_CUPTI_DEBUG and CHARM_LB_CUPTI_TIME are left alone.
Builds clean, with no unused-variable warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Migrating a chare's device state stages it into a fresh allocation, which has to outlive the send because the transport may still be reading it. That was arranged by hand: the sender kept the entry in sendGPUBuffers, and the destination sent a finishGPUSend entry method back once it had unpacked, which freed the block. The receiver had to remember where the payload came from to address that message, which is what receivedDeviceSrcPe and the srcPe parameter on immigrateGPU were for. The zerocopy layer already reports exactly this event. A CkDeviceBuffer's source callback is defined as the signal that the source buffer may be reused, and it is deliberately never aggregated -- deviceMintAck marks an id carrying one as urgent so the receiver returns it immediately rather than holding it for the piggyback, precisely because a sender is waiting on it. So hand the buffer its own release: CkDeviceBuffer(gpuData.data, CkCallback(gpuMigrateStagedFree, gpuData.data)) CkCallback(CkCallbackFn, param) records the PE that built it and routes back there, so this runs on the sending PE -- the context finishGPUSend ran in -- and param is the staged pointer, so nothing has to be looked up. The map entry is erased as soon as the send is issued; the callback owns the block from then. That removes finishGPUSend (definition, declaration and entry), the receivedDeviceSrcPe map, the gpuSrcPe plumbing, and the now-unused srcPe parameter on both immigrateGPU variants. Checked that the callback fires on every path, since a dropped one would be a silent device leak rather than a crash: src_cb is recorded unconditionally in CkRdmaDeviceIssueRgets before any mode branching; intra-node transfers complete on the stream and invoke it from the receive handler; inter-node ones delete the receiver's copy on purpose because the sender minted an ack id and fires the callback itself when the ack returns. Compile-verified only. Not run: the failure mode here is a leak, which no build will show, so this wants a GPU migration run before it is trusted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
adityapb
marked this pull request as ready for review
September 11, 2026 14:06
adityapb
requested review from
ericjbohm,
lvkale and
ritvikrao
as code owners
September 11, 2026 14:06
The timeline sweep kept the kernels currently on the device in a std::set<int, decltype(cmpActive)>, ordered by start time then index. The comment said that was so iteration order would be FIFO by submission. Nothing depends on that order. The set is iterated exactly twice per interval: once to sum sms_used into `want`, and once to credit each kernel dt_s * used/want. A sum and a set of independent per-element updates give the same answer in any order -- nothing reads a first element, stops early or breaks on a condition. The ordering is a leftover from a design that was deliberately removed. The comment directly above the split argues against handing SMs out FIFO by submission and stopping at the first kernel that finds the pool empty, because whether an object is credited would then depend on where its kernels land in the launch order, which reshuffles whenever placement changes. Proportional sharing replaced it, and the comparator outlived it -- so the container's stated purpose contradicted the reasoning a few lines below it. Use a vector reserved to the kernel count, with swap-and-pop erase. That drops O(log n) insert/erase with node allocation, and a comparator that dereferenced `kernels` on every comparison, for contiguous iteration -- on the inner loop of the sweep, two events per kernel. #include <set> goes with the last std::set in the file. Results are unchanged, including bitwise: `want` is a long, so that sum is exact in any order, and demand accumulates per kernel rather than across the set, so no floating-point sum is reordered. Not measured. The tracing itself is the dominant cost of GPU load measurement, so this is unlikely to be visible next to it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…LBLoadMigBal" This reverts commit d3d9cf7. +LBLoadMigBal was the wrong lever for the problem it was aimed at. It governs selection BETWEEN finished candidates: among those inside the migration budget, one with fewer migrations wins only if its max load is within the factor of the best. The knob the balancer already has for this tradeoff is A, which governs what each candidate is willing to accept in the first place -- M *= A sets the target max load as a multiple of what plain greedy achieves, and the refine loop leaves an object where it is whenever its PE stays under M. A is the tolerance the manual documents for GreedyRefineLB: "the tolerance it should allow above the maximum load Greedy would produce (e.g. 1.1 allows the maximum load to be 10% higher than Greedy's max load)". Acting there is both the documented interface and the better-aimed one, since it changes the stay-put decision itself rather than re-ranking results after the fact. The measurement in the reverted commit still stands -- the default does move nearly every object for a fraction of a percent of max load -- but the fix belongs at the tolerance, which the next commit exposes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The manual documents a tolerance for the greedy-refine balancers: how far above the max load plain greedy produces the balancer may go, in exchange for migrating less. GreedyRefineCentralGPULB had the parameter (A, applied as M *= A) but no way to set it, because both constructors forced concurrent mode on, which overwrites A per-PE from a rank table. So the balancer always searched for a tolerance and could never be told one. +LBGreedyRefineTolerance sets it. The default is 1.1: aim within 10% of greedy's max load and spend the slack on not migrating. 0 or less asks for the old search instead, which stays reachable but is no longer the default -- it costs a reduction per balancing step and only ever samples as many of the 225 (A,B) pairs as there are PEs. Setting the tolerance alone did nothing, which the first run showed plainly: every value from 1.05 to 1.5 gave zero migrations and an identical result. The two decisions this balancer makes were not guarded alike. Choosing a GPU group tests both a relative band (B) and a ceiling (M). Choosing a PE within that group tested only B, so with B = FLT_MAX -- the long-dead non-concurrent default -- it was unconditionally true and nothing ever moved. maxCpuLoad was computed but only ever reported, never consulted. So build Mcpu, the host-side twin of M: the same greedy trial over per-PE host loads, scaled by the same A, and require that a PE absorb the object and stay under it before the object is left in place. It ratchets upward when exceeded, exactly as M does, so neither is a hard ceiling. Measured, jacobi2d-imbalance, 256 chares on 32 PEs, 3 LB steps, one A40 node: search (old default) 366 moves 0.136 s tolerance 1.05 47 moves 0.064 s tolerance 1.1 26 moves 0.060 s tolerance 1.2 0 moves 0.048 s tolerance 1.5 0 moves 0.047 s Monotonic, where before the fix all of them were 0. The +LBDebug line confirms it binds for the right reason: at tolerance 1.05 the achieved host max is 1.07-1.14x greedy's, not the 19x the earlier run reported. That diagnostic is restored from stock, which the GPU variant had dropped with (void) casts, and split per dimension. One combined ratio was unreadable here: it divided the host max by the GPU greedy max, and this application's measured GPU load is near zero, which is where 19x came from. Caveat: the searched path now carries a constraint it did not have, so its behaviour changes too. Baseline moves varied 366-545 across runs of the same configuration, so these are single runs on a small workload. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ck.h includes "cldb.h" under CMK_USE_SHMEM on the classic Converse path, but the header was never added to conv-core-h-to-install, so a non-reconverse build had the include with nothing to satisfy it. Reconverse builds are unaffected: they get reconverse's own src/cldb.h, and the copy that lands in the build's include directory is byte-identical to it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lvkale
reviewed
Sep 12, 2026
lvkale
left a comment
lvkale
left a comment
Contributor
There was a problem hiding this comment.
Read the load-balancer, migration and device-zerocopy parts against reviewed-with-reconverse. Nothing here should hold the merge; five comments, in order of how much I would like them addressed before it lands.
1. Base update. The branch is still on 310d470e0, before #3974 bumped contrib/reconverse (reconverse #217, #222) and widened reconverse-ci's tier to fourteen directories. GitHub does not rerun checks when the base moves, so the green checks describe the old reconverse and the five-directory tier. Please rebase or merge the base so they run against what this will actually land on; there are no conflicting files.
2. The CMK_GLOBAL_LOCATION_UPDATE dependence, and a suggestion for the follow-up. As I read ckrdmadevice.C, the reason is the shape of the device transfer, not the balancer: the metadata is an ordinary array message and gets forwarded to the element's current PE, but the payload is addressed at send time to dest_pe (CmiSendDevice(dest_pe, ptr, cnt, tag) under CMK_GPU_COMM, the IPC/staged path otherwise, with the explicit abort at ckrdmadevice.C:258 when source.dest_pe != CkMyPe()). A migration between the sender's lookup and delivery strands the payload at the old PE; under CMK_GPU_COMM that is an unmatched tagged send rather than an abort, i.e. a hang. The global update narrows the window (and the option's own rule confines migrations to LB steps) but a send in flight during the step still targets the old PE. For the PR that removes the dependence, the host zerocopy entry-method API's design fits: make the payload receiver-driven, so the element's PE, wherever the metadata reaches it, issues a device get from the source PE, which has not moved. Then no location knowledge is needed at the sender at all, and migrateMe() comes back for GPU applications. If that is already the plan, ignore this; if not, please consider it before another mechanism grows around the current one.
3. conv-mach-cuda.sh now links -lcupti and defines HAPI_CUPTI_LB unconditionally for every legacy (buildold) CUDA build, while the cmake path defines HAPI_CUPTI_LB only when it finds the CUPTI library. A CUDA installation without extras/CUPTI builds with cmake and fails to link with the legacy script. Guarding the shell side the same way (test for $CUDA_DIR/extras/CUPTI/lib64) keeps the two builds consistent.
4. AMD: the migration half could be portable today. pupDeviceCopy in pup_util.C is a raw cudaMemcpy under #if CMK_CUDA, so a HIP build has no device-mode pup at all, although everything around it is already portable (hapiMalloc staging, and the device zerocopy transport that ran on MI250X). Routing that copy through HAPI's device-to-device memcpy, and the gpumigrate test's setup copies likewise, makes device-state migration work on AMD independently of measurement. The measurement half stays CUDA-only until someone writes the roctracer/rocprofiler collector, which the HIP warning already says well. The manual's "CUDA only" note would then distinguish the two.
5. Manual note. The requirement paragraph should say the dependence is temporary and why (comment 2), so users do not design the AtSync-only restriction into applications as permanent.
Two things that read as clearly right: the CentralLB change not updating the source PE's own cache before it emigrates, which fixes a real first-migration assert in the existing option's path independent of GPUs; and gating the pup surface through a CI-compiled test with poisoned destination buffers and a deliberately misaligned second buffer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GPU-aware load balancing and device-state migration
Lets a GPU application be balanced on the measured GPU cost of each object, and lets a chare whose state lives in device memory migrate with that state intact. Three pieces that only make sense together: measure the device work, have a strategy that can act on it, and be able to move an object once the strategy decides to.
Measurement
CUPTI activity tracing attributes each kernel to the object that launched it. The runtime already brackets every entry method, so applications need no annotation unless they launch outside one (
CUPTI_LAUNCH_WRAPPER).Per-kernel records are reduced, while still process-local, to one scalar per object, and only those scalars reach the central balancer. The reduction is a sweep over the device timeline that splits each interval among the kernels holding the device in proportion to the SMs each occupies, so the result is seconds of whole-device occupancy rather than wall time. That matters for placement: it estimates what an object would cost on a different device, not only what it cost sharing this one.
Measurement follows
LBTurnInstrumentOn()/LBTurnInstrumentOff(), so an application that instruments a window around its ownAtSyncpays the tracing cost only inside it. Tracing is the dominant cost of the feature.Strategy
GreedyRefineCentralGPULBbalances at two granularities at once. Objects are assigned across GPU groups — the PEs sharing a device — by GPU load, then across PEs within a group by host load. This matches the usual one-GPU-per-process arrangement: device work decides which GPU an object runs on, host work decides which PE drives it.Migration
Device buffers are pup'd with
PUPMode::DEVICE. On emigration the state is copied into a staging allocation so the element can be destroyed immediately, shipped over the device zerocopy path as anocopydeviceentry-method argument, and rebound on arrival. The staging buffer is released by the buffer's own source callback, which is the zerocopy layer's existing signal that a source buffer may be reused, rather than by a hand-rolled ack message from the destination.Prerequisite, and a restriction worth reading
Requires
CMK_GLOBAL_LOCATION_UPDATE. A device zerocopy send is addressed to the PE the sender believes hosts the target, so without it a send issued around a migration reaches a PE that no longer hosts the element, and the receive path aborts. The option brings its own restriction: migrations must happen at load balancing steps, so a GPU application migrates throughAtSync(), nevermigrateMe().Tuning:
+LBGreedyRefineToleranceAmong candidate assignments within the migration budget, one with fewer migrations wins only if its max load is within this factor of the best. It was a hardcoded 1.003, tight enough that it rarely binds — and since PE 0 deliberately runs plain greedy with migrations ignored, the winner is usually that one, which moves nearly every object to gain a fraction of a percent.
Measured on
jacobi2d-imbalance, 256 chares on 32 PEs, 3 LB steps, one A40 node, varying only this value:+LBGreedyRefineToleranceThe default is left unchanged here: one small workload is not enough to re-pick it, and the knob now makes that cheap to investigate.
Platform
CUDA only. Every CUPTI-specific declaration, definition and struct field is behind
CMK_CUDA. On HIP there is no equivalent measurement wired up, so the balancer warns at construction that it will see zero GPU load and balance on host load alone, rather than appearing to balance on a dimension that is empty.Note
HAPI_CUPTI_LBis not a second spelling ofCMK_CUDA: cmake defines it only when it finds the CUPTI library, so a CUDA build without CUPTI has one and not the other, and the inner gate and its stubs exist for that case.Testing
tests/charm++/cuda/gpumigratecovers device-state migration: two buffers per element, the second deliberately not a multiple ofDEVICE_PUP_ALIGNso that a packer and unpacker disagreeing about where it starts cannot go undetected; destination buffers poisoned before the device pup so the test cannot pass by reading memory the allocator happened to hand back; driven throughAtSyncwithRotateLB, since no load-based strategy would move identically loaded objects. CI compiles it, which gates thePUPMode::DEVICEsurface and thenocopydeviceentry method; running it needs a GPU.