prov/efa: dmabuf try / fallback logic #11465
Open
+215
−98
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
shijin-aws
reviewed
Oct 4, 2025
shijin-aws
reviewed
Oct 4, 2025
jiaxiyan
requested changes
Oct 8, 2025
nmazzilli3
force-pushed
the
Subspace-2834
branch
from
c1be6c7 to
d626afc
Compare
shijin-aws
reviewed
Oct 13, 2025
nmazzilli3
force-pushed
the
Subspace-2834
branch
2 times, most recently
from
4d7e686 to
8901eae
Compare
jiaxiyan
reviewed
Oct 23, 2025
nmazzilli3
force-pushed
the
Subspace-2834
branch
from
8901eae to
44d02b1
Compare
jiaxiyan
reviewed
Nov 11, 2025
jiaxiyan
reviewed
Nov 11, 2025
feat: Add dmabuf capability tracking for HMEM interfaces Problem: - Need infrastructure to detect and query dmabuf support per HMEM interface - Providers need to know if dmabuf is available before attempting registration Solution: - Added ofi_hmem_is_dmabuf_supported() query function in include/ofi_hmem.h - Implemented dmabuf support detection in src/hmem_cuda.c - Implemented dmabuf support detection in src/hmem_neuron.c - Added stub implementations in src/hmem_rocr.c and src/hmem_synapseai.c - Updated src/hmem.c to expose the new interface Testing: - Verified compilation with and without CUDA/Neuron support - Tested query function returns correct values for each interface Sim Issue: - N/A Signed-off-by: Nick Mazzilli <[email protected]>
nmazzilli3
force-pushed
the
Subspace-2834
branch
from
44d02b1 to
002320b
Compare
feat: Add default dmabuf attempt with fallback for all efa_hmem_ifaces Problem: - Need to make dmabuf usage default going forward - Need fallback mechanism when dmabuf is not supported or fails Solution: - Modified initial PR from @jiaxiyan at ofiwg@6aa6708 - Added dmabuf_supported_by_device_b flag in efa_hmem_info structure in prov/efa/src/efa_hmem.h - Updated dmabuf_supported_by_device_b detection in each fi_hmem_iface p2p_support function in prov/efa/src/efa_hmem.c - Modified efa_mr_reg_ibv_mr() in prov/efa/src/efa_mr.c to use efa_mr->peer.iface for dmabuf checks - Implemented try-dmabuf-first with fallback to ibv_reg_mr in efa_mr_reg_ibv_mr() - Added environment variable control for dmabuf enable/disable per interface Testing: - Ran MPI perf tests on 2 nodes on p5en with dmabuf and fallback - Ran MPI perf tests on 16 nodes on p5en with dmabuf and fallback - Verified fallback works when dmabuf is unavailable Sim Issue: - N/A Signed-off-by: Nick Mazzilli <[email protected]>
nmazzilli3
force-pushed
the
Subspace-2834
branch
from
002320b to
8e504f6
Compare
nmazzilli3
commented
Nov 12, 2025
jiaxiyan
reviewed
Nov 12, 2025
| info->max_medium_msg_size = 0; | ||
| info->runt_size = 0; | ||
| info->min_read_msg_size = 0; | ||
| info->min_read_write_size = 0; |
Contributor
There was a problem hiding this comment.
These are default to 0 and set in efa_domain_hmem_info_init_protocol_thresholds.
| info->dmabuf_supported_by_device_b = true; | ||
| } else { | ||
| EFA_INFO(FI_LOG_CORE, "FI_HMEM_SYNAPSEAI DMABUF disabled by environment variable\n"); | ||
| info->dmabuf_supported_by_device_b = false; |
Contributor
There was a problem hiding this comment.
You don't need to set it again here
Contributor
j-xiong
reviewed
Nov 18, 2025
| * @param[in] iface The HMEM interface to check | ||
| * @return true if DMABUF is enabled for the interface, false otherwise | ||
| */ | ||
| bool efa_hmem_is_dmabuf_env_var_enabled(enum fi_hmem_iface iface) |
Contributor
There was a problem hiding this comment.
Maybe we should add a method to the common hmem interface and define an ofi_hmem_is_dmabuf_requested function in src/hmem.c. That make the code reusable for other providers. Also need to add the function for FI_HMEM_ZE is going this route.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
feat: Adding default dmabuf attempt and fallback logic for all efa_hmem_ifaces
Problem:
Solution:
Testing:
Sim Issue: