Add CUDA forward compatibility hook #906
Conversation
elezar
force-pushed
the
add-compat-lib-hook
branch
from
f21f7a6 to
d617fb5
Compare
|
|
||
| // Create the 'compat-libs' command | ||
| c := cli.Command{ | ||
| Name: "compat-libs", |
There was a problem hiding this comment.
Should this hook have a better name?
Currently it reads:
nvidia-cdi-hook compat-libs --driver-version 999.88.77
There was a problem hiding this comment.
Changed this to:
nvidia-cdi-hook cuda-compat --host-driver-version=999.88.77
We could use this hook for other compat functions such as enhanced compatibility.
elezar
force-pushed
the
add-compat-lib-hook
branch
3 times, most recently
from
606aed5 to
ba30397
Compare
elezar
force-pushed
the
add-compat-lib-hook
branch
3 times, most recently
from
0c4dd3d to
0b50458
Compare
elezar
added
the
must-backport
elezar
force-pushed
the
add-compat-lib-hook
branch
2 times, most recently
from
d1e97a0 to
eeef750
Compare
elezar
force-pushed
the
add-compat-lib-hook
branch
2 times, most recently
from
88841fa to
8e0c69b
Compare
internal/modifier/gated.go
Outdated
| compatLibHookDiscoverer := discover.NewCUDACompatHookDiscoverer(logger, cfg.NVIDIACTKConfig.Path, driver) | ||
| discoverers = append(discoverers, compatLibHookDiscoverer) | ||
| if cfg.NVIDIAContainerRuntimeConfig.Mode == "legacy" { | ||
| ldcacheIpdateHookDiscoverer, err := discover.NewLDCacheUpdateHook( |
There was a problem hiding this comment.
Note: Since the nvidia-container-runtime-hook is invoked as a prestart hook, this is done BEFORE the createContainer hook that we insert above. This means that we need to once again run the update ldcache hook.
There was a problem hiding this comment.
@klueska one option would be to create the file in /etc/ld.so.conf.d when invoking the nvidia-container-runtime-hook instead -- before calling out to the nvidia-container-cli. We don't have as ready access to the driver version, but we could extract it there.
There was a problem hiding this comment.
If we create the file in nvidia-container-runtime-hook, then wouldn't that remove the need to add the cuda-compat createContainer hook altogether? And as a result, the forward compatibility support in the legacy stack would not require users to use NVIDIA container Runtime, correct?
There was a problem hiding this comment.
I did try it, and my initial assumption was incorrect. Because we invoke ldconfig as:
argv = (char * []){cnt->cfg.ldconfig, "-f", "/etc/ld.so.conf", "-C", "/etc/ld.so.cache", cnt->cfg.libs_dir, cnt->cfg.libs32_dir, NULL};
in libnvidia-container, this means that the cnt->cfg.libs_dir and cnt->cfg.libs32_dir folders take precedence over the files in /etc/ld.so.conf.d and the CUDA libraries present there are used.
We could rework the libnvidia-container implementation further, but the intent of this change is also to provide the functionality for #910 so that we can remove the legacy code path by default.
elezar
force-pushed
the
add-compat-lib-hook
branch
3 times, most recently
from
77bfb3c to
1140fcf
Compare
elezar
force-pushed
the
add-compat-lib-hook
branch
from
1140fcf to
cb6b4f6
Compare
elezar
force-pushed
the
add-compat-lib-hook
branch
2 times, most recently
from
f39a194 to
aaf5ed8
Compare
This change adds an nvidia-cdi-hook cuda-compat hook that checks the container for cuda compat libs and updates /etc/ld.so.conf.d to include their parent folder if their driver major version is sufficient. This allows CUDA Forward Compatibility to be used when this is not available through the libnvidia-container. Signed-off-by: Evan Lezar <[email protected]>
This change adds the cuda-compat hook to the incomming OCI runtime spec if the allow-cuda-compat-libs-from-container feature flag is not enabled. An update-ldcache hook is also injected to ensure that the required folders are processed. Signed-off-by: Evan Lezar <[email protected]>
Signed-off-by: Evan Lezar <[email protected]>
Signed-off-by: Evan Lezar <[email protected]>
Signed-off-by: Evan Lezar <[email protected]>
Signed-off-by: Evan Lezar <[email protected]>
Signed-off-by: Evan Lezar <[email protected]>
Signed-off-by: Evan Lezar <[email protected]>
elezar
force-pushed
the
add-compat-lib-hook
branch
from
aaf5ed8 to
12cc9aa
Compare
With #877 the default behaviour of the NVIDIA Container Runtime / NVIDIA Container Runtime Hook was changed to not mount compat libraries from the container into the container. This removed "automatic" support for CUDA Forward compatibility.
This change attempts to address this by adding a
createContainerHookthat will create a file in/etc/ld.so.conf.d/in the container to ensure that the/usr/local/cuda/compatlibraries are added to the ldcache over the libraries mounted from the host. Here the provided driver version number is used to indicate whether this should be done or not.In the case of the legacy runtime, this behaviour is only triggered if the
allow-cuda-compat-libs-from-containerfeature flag is not enabled. The CDI spec generation has also been extended to include this hook.