Add host memory limits by nirandaperera · Pull Request #1193 · rapidsai/rapidsmpf

nirandaperera · 2026-09-01T22:33:09Z

Pageable-host allocations were not tracked, and pinned-pool limits were not reflected in reservation accounting, allowing reservations beyond the configured pool maximum. This adds the host-memory accounting and limit handling needed for disk-spill decisions.

Generalize the resource adaptor to track pageable-host allocations and include them in statistics.
Add an absolute spill_host_limit, unbounded by default, and derive the pageable-host budget after accounting for bounded pinned memory.
Fix pinned-memory configuration by wiring pinned_max_pool_size into reservation limits, supporting disabled as unbounded, and rejecting zero or incompatible limits.
Update Python bindings, documentation, and C++/Python test coverage.

This is related to #1170. IMO it would be a nice to have change in general.

Option Defaults

Option	Default	Effect
`pinned_memory`	false	No pinned pool. PINNED_HOST limit is 0.
`pinned_initial_pool_size`	0%	Used only if pinned is on.
`pinned_max_pool_size`	80%	Used only if pinned is on. 80% of per-GPU host memory.
`spill_host_limit`	disabled	Unbounded pageable HOST.

Effect on PINNED_HOST and HOST

Following table shows the effect on memory limits based on different option settings.

`pinned_memory`	`pinned_max_pool_size`	`spill_host_limit`	`PINNED_HOST` limit	`HOST` limit	Checks
off	-	disabled	`0`	unlimited
off	-	set	`0`	set
on	disabled	disabled	unlimited	unlimited
on	disabled	set	unlimited	set
on	set	disabled	set	unlimited	`pinned_max <= numa_host`
on	set	set	set	set	`pinned_max <= numa_host`, then `spill_host_limit <= total_numa_host - pinned_max`

numa_host = host memory of current numa node
total_numa_host = host memory of current numa nodes (governed by numa policy setting)
If pinned_max_pool_size is set, it should be greater-than zero

Indicative values

	GB200	DGX B200	g7e.8xlarge
host mem/node	960GB	2TB	256 GB
GPUs/node	4	8	1
host mem/GPU	240GB	256GB	256GB
80% host mem/GPU	192GB	204GB	204GB

Ideally, pinned_max_pool_size + spill_host_limit <= 80% host_mem_per_gpu

- Track pageable-host allocations through the generalized resource adaptor. - Add an absolute, unbounded-by-default host memory limit. - Reconcile pageable and pinned-host budgets and reject invalid pinned limits. - Extend statistics, Python bindings, documentation, and tests. Signed-off-by: niranda perera <niranda.perera@gmail.com>

…-limits

pentschev · 2026-09-07T17:30:15Z

+ * The limit must be an absolute byte count. Disabled values produce an
+ * unbounded pageable-host budget. This limit is independent of
+ * `pinned_max_pool_size`. When both limits are bounded,
+ * `BufferResource::from_options()` rejects configurations where their sum
+ * exceeds the summed host memory of the nodes in the calling thread's memory
+ * policy. The pinned maximum is also constrained by the host memory of its
+ * NUMA node.


So now users have two limits they have to independently set not to cause a host OOM? If so, I don't think this is a good UX, it will be difficult for users to ensure they're both within safe limits.

Also, while the sum of the two is in theory correct, in practice this is an unfeasible limit, since OS alone takes non-negligible amounts of memory. I don't know what would be a safe value, but perhaps looking at the amount of free memory when the process is starting minus some percentage is more reasonable.

I see your point.
One issue I had was, what should be the max limit for HOST memory?
In multi GPU nodes, I think its correct to limit PINNED to host_memory_per_gpu because a pool can only be assigned to a single numa ID. But HOST can span across numa nodes.
But since we are running with rrun (or with apply_bindings), I think the processes will have MPOL_BIND set to proper numa nodes.
So, in that case, HOST + PINNED < get_numa_node_host_memory(get_current_numa_node()) IINM.

Getting free memory is a reasonable idea. let me look at that

pentschev · 2026-09-07T17:39:49Z

+        if (host_limit != unbounded && pinned_limit != unbounded) {
+            auto const numa_host = safe_cast<std::int64_t>(
+                get_numa_node_host_memory(pinned_pool_properties->numa_id)
+            );
+            RAPIDSMPF_EXPECTS(
+                pinned_limit <= numa_host,
+                "pinned_max_pool_size exceeds NUMA node host memory",
+                std::invalid_argument
+            );


Could we move the pinned_limit <= numa_host validation outside this combined bounded-host/bounded-pinned condition, and mirror that change in the Python from_options() path? As written, pinned_max_pool_size can be set above its NUMA node’s host memory whenever spill_host_limit is left at its default "disabled". That contradicts the new configuration documentation and accepts an incompatible pool configuration, only the combined pageable+pinned budget check needs both limits to be bounded. Please add also coverage for a bounded oversized pinned limit with spill_host_limit=disabled.

@pentschev I am not sure if I understood you correctly. spill_host_limit=="disabled" implies that host memory is unbounded. if pinned_pool_max_size is not set, but spill_host_limit is set, we cant reliably ensure host_limit <= total_host - pinned_limit during BR creation.

pentschev · 2026-09-07T18:43:48Z

+  - **Description**: Soft upper limit on pageable host memory configured for
+    RapidsMPF, independent of `pinned_max_pool_size`. When both limits are
+    bounded, their sum cannot exceed the summed host memory of the nodes in the
+    calling thread's memory policy; the pinned maximum also cannot exceed its
+    NUMA node's host memory. This is a coarse validation because other processes
+    and allocations may consume the same host memory. It accepts absolute byte
+    counts (for example, `"10GiB"` or `"512MB"`). Percentages are not supported
+    because the appropriate host-memory share depends on the job's process and
+    NUMA topology.


This looks closer to proper docstrings than that from host_limit_from_options. However, those same issues are still relevant, perhaps taking into account the free memory is more realistic. I also don't understand how depending on job's processes/NUMA topology voids the potential for using percentages.

pentschev · 2026-09-07T19:06:37Z

        # means pinned host memory is disabled.
        cdef optional[cpp_PinnedPoolProperties] props = \
            pinned_pool_properties_from_options(options._handle)
+        unbounded = 2**63 - 1


from libc.stdint cimport INT64_MAX, int64_t ... cdef int64_t unbounded = INT64_MAX

…-limits

Signed-off-by: niranda perera <niranda.perera@gmail.com>

…-limits

nirandaperera · 2026-09-09T05:34:34Z

@pentschev I thought the spill_host_limit value representation a bit more.

I think we can not reliably use a % of available/ free memory. It may give use an earlier signal before an OOM, but

This changes the current default behavior
In a multiprocess run, % of free memory is always a moving target during startup, unless some process determines this and broadcasts.

So, I decided to leave spill_host_limit as a value-only option (no percentages), that defaults to disabled (to preserve the current default behavior). I updated the PR description to explain the all option permutations and the checks that were put in place.

nirandaperera self-assigned this Sep 1, 2026

nirandaperera requested review from a team as code owners September 1, 2026 22:33

nirandaperera added improvement Improves an existing functionality non-breaking Introduces a non-breaking change labels Sep 1, 2026

nirandaperera changed the title ~~Add host memory limits for disk spilling~~ Add host memory limits Sep 1, 2026

nirandaperera mentioned this pull request Sep 3, 2026

Add a file-backed DiskBuffer #1195

Draft

nirandaperera added 2 commits September 3, 2026 11:01

Merge branch 'main' of github.com:rapidsai/rapidsmpf into host-memory…

e1566bd

…-limits

nirandaperera force-pushed the host-memory-limits branch from f0bf4cb to e1566bd Compare September 3, 2026 18:01

nirandaperera mentioned this pull request Sep 5, 2026

Disk-spilling shuffler #1170

Open

pentschev requested changes Sep 7, 2026

View reviewed changes

nirandaperera added 4 commits September 8, 2026 09:06

Merge branch 'main' of github.com:rapidsai/rapidsmpf into host-memory…

db41755

…-limits

minor improvements

b7bc05b

Signed-off-by: niranda perera <niranda.perera@gmail.com>

streamline pinned and host memory limit value verification

8e835c5

Signed-off-by: niranda perera <niranda.perera@gmail.com>

Merge branch 'main' of github.com:rapidsai/rapidsmpf into host-memory…

a7a8aff

…-limits

nirandaperera requested a review from pentschev September 9, 2026 05:34

Navigation Menu

Uh oh!

Add host memory limits - #1193

Add host memory limits #1193
nirandaperera wants to merge 6 commits into
rapidsai:mainfrom
nirandaperera:host-memory-limits

nirandaperera commented Sep 1, 2026 •

edited

Loading

Uh oh!

pentschev Sep 7, 2026

Uh oh!

nirandaperera Sep 8, 2026

Uh oh!

pentschev Sep 7, 2026

Uh oh!

nirandaperera Sep 9, 2026

Uh oh!

pentschev Sep 7, 2026

Uh oh!

pentschev Sep 7, 2026

Uh oh!

nirandaperera commented Sep 9, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

nirandaperera commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Option Defaults

Effect on PINNED_HOST and HOST

Indicative values

Uh oh!

pentschev Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

nirandaperera Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

pentschev Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

nirandaperera Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

pentschev Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

pentschev Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

nirandaperera commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nirandaperera commented Sep 1, 2026 •

edited

Loading

nirandaperera commented Sep 9, 2026 •

edited

Loading