From 88543d9081e6b99a42288d6342658956f971ce21 Mon Sep 17 00:00:00 2001 From: "Carson M." Date: Tue, 11 Mar 2025 00:10:31 -0500 Subject: [PATCH] docs: update EP dynamic linking info --- docs/pages/perf/execution-providers.mdx | 29 ++++++++++++++----------- docs/pages/setup/linking.mdx | 5 ++--- 2 files changed, 18 insertions(+), 16 deletions(-) diff --git a/docs/pages/perf/execution-providers.mdx b/docs/pages/perf/execution-providers.mdx index fec57da2..ec1eba0a 100644 --- a/docs/pages/perf/execution-providers.mdx +++ b/docs/pages/perf/execution-providers.mdx @@ -11,12 +11,11 @@ ONNX Runtime must be compiled from source with support for each execution provid * 🔷 - Supported by `ort`. * ✅ - Static binaries provided by pyke. -* 🟢 - Dynamic binaries provided by pyke. | EP | Cargo feature | Supported | Binaries | |:-------- |:------- |:-------:|:------:| -| [NVIDIA CUDA](https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html) | `cuda` | 🔷 | 🟢 | -| [NVIDIA TensorRT](https://onnxruntime.ai/docs/execution-providers/TensorRT-ExecutionProvider.html) | `tensorrt` | 🔷 | 🟢 | +| [NVIDIA CUDA](https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html) | `cuda` | 🔷 | ✅ | +| [NVIDIA TensorRT](https://onnxruntime.ai/docs/execution-providers/TensorRT-ExecutionProvider.html) | `tensorrt` | 🔷 | ✅ | | [Microsoft DirectML](https://onnxruntime.ai/docs/execution-providers/DirectML-ExecutionProvider.html) | `directml` | 🔷 | ✅ | | [Apple CoreML](https://onnxruntime.ai/docs/execution-providers/CoreML-ExecutionProvider.html) | `coreml` | 🔷 | ✅ | | [AMD ROCm](https://onnxruntime.ai/docs/execution-providers/ROCm-ExecutionProvider.html) | `rocm` | 🔷 | ❌ | @@ -32,17 +31,9 @@ ONNX Runtime must be compiled from source with support for each execution provid | [AMD MIGraphX](https://onnxruntime.ai/docs/execution-providers/MIGraphX-ExecutionProvider.html) | `migraphx` | 🔷 | ❌ | | [AMD Vitis AI](https://onnxruntime.ai/docs/execution-providers/Vitis-AI-ExecutionProvider.html) | `vitis` | 🔷 | ❌ | | [Rockchip RKNPU](https://onnxruntime.ai/docs/execution-providers/community-maintained/RKNPU-ExecutionProvider.html) | `rknpu` | 🔷 | ❌ | -| WebGPU | `webgpu` | 🔷 | ✅\* | +| WebGPU | `webgpu` | 🔷 | ✅ | | [Microsoft Azure](https://onnxruntime.ai/docs/execution-providers/Azure-ExecutionProvider.html) | `azure` | 🔷 | ❌ | -
-

\* WebGPU binaries are currently only available for wasm32-unknown-emscripten.

-
- - - Note that a few execution providers (notably CUDA, TensorRT, and ROCm) are only available with **dynamic linking**. Dynamic linking may require additional setup to get working; see [Linking > Dynamic linking](/setup/linking#dynamic-linking). - - ## Registering execution providers To use an execution provider with `ort`, you'll need to enable its respective Cargo feature, e.g. the `cuda` feature to use CUDA, or the `coreml` feature to use CoreML. @@ -213,11 +204,18 @@ If it seems like the execution provider is not registering properly, or you are ## Notes +### Dynamically-linked EP requirements +Certain EPs like CUDA and TensorRT use a separate interface that require them to be compiled as dynamic libraries which are loaded at runtime when the EP is registered. The DirectML and WebGPU EP do not use this interface, but do require helper dylibs. + +Due to the quirks of dynamic library loading, you may encounter issues with builds including these EPs due to ONNX Runtime failing to find the dylibs at runtime. `ort`'s `copy-dylibs` Cargo feature (which is enabled by default) tries to alleviate this issue by symlinking these dylibs into your `target` folder so they can be found by your application when in development. On Windows platforms that don't have [Developer Mode](https://learn.microsoft.com/en-us/windows/uwp/get-started/enable-your-device-for-development) enabled, a copy is instead performed (excluding examples and tests). On other platforms, additional setup is required to get the application to load dylibs from its parent folder. + +See [Runtime dylib loading](/setup/linking#runtime-dylib-loading) for more information. + ### CUDA `ort` provides binaries for CUDA 12 with cuDNN 9.x only. Make sure the correct version of CUDA & cuDNN are installed and available on the `PATH`. ### CoreML -Statically linking to CoreML (the default behavior when using downloaded binaries + the `coreml` Cargo feature) requires an additional Rust flag in order to link properly. You'll need to provide the flag `-C link-arg=-fapple-link-rtlib` to `rustc`. You can do this via a build script, an entry in [`.cargo/config.toml`](https://doc.rust-lang.org/cargo/reference/config.html), or an environment variable. +Statically linking to CoreML (the default behavior when using downloaded binaries) requires an additional Rust flag in order to link properly. You'll need to provide the flag `-C link-arg=-fapple-link-rtlib` to `rustc`. You can do this via a build script, an entry in [`.cargo/config.toml`](https://doc.rust-lang.org/cargo/reference/config.html), or an environment variable. @@ -249,3 +247,8 @@ Statically linking to CoreML (the default behavior when using downloaded binarie ``` + +### WebGPU +The WebGPU EP is **experimental** and may produce incorrect results/crashes; these issues should be reported upstream as there's unfortunately nothing we can do aobut them. + +WebGPU binaries are provided for Windows & Linux. On Windows, the build supports running on DirectX 12 or DirectX 11. On Linux, it supports Vulkan & OpenGL/GLES. diff --git a/docs/pages/setup/linking.mdx b/docs/pages/setup/linking.mdx index f806342d..27ee0a11 100644 --- a/docs/pages/setup/linking.mdx +++ b/docs/pages/setup/linking.mdx @@ -18,8 +18,6 @@ $ ORT_LIB_LOCATION=~/onnxruntime/build/Linux/Release cargo build For iOS (or for other platforms if you are compiling multiple profiles at once), you'll need to manually specify the profile with the `ORT_LIB_PROFILE` environment variable. If not specified, `ort` will prefer `Release` over `RelWithDebInfo` over `MinSizeRel` over `Debug`. ## Dynamic linking -Some execution providers unfortunately only support dynamic linking. Dynamic linking doesn't play well with the Rust ecosystem, though `ort` tries to alleviate the pain as much as possible. - When it comes to dynamic linking, there are two options: `load-dynamic`, or standard compile-time dynamic linking. We recommend `load-dynamic` as it gives more control and is often far less troublesome to work with. ### Runtime loading with `load-dynamic` @@ -71,7 +69,8 @@ ort = { version = "2", features = [ "load-dynamic" ] } ### Compile-time dynamic linking For compile-time dynamic linking, you'll need to configure your environment in the exact same way as if you were [statically linking](#static-linking). -Note that the dylibs then have to be placed in a certain location for them to be found by the executable. For Windows, this is either somewhere on the `PATH`, or in the same folder as the executable. On macOS and Linux, they have to be placed somewhere in the `LD_LIBRARY_PATH`, or you can use rpath to configure the executable to search for dylibs in its parent folder. We've had the least issues with rpath, but YMMV. +#### Runtime dylib loading +Dylibs linked at compile-time need to be placed in a specific location for them to be found by the executable. For Windows, this is either somewhere on the `PATH`, or in the same folder as the executable. On macOS and Linux, they have to be placed somewhere in the `LD_LIBRARY_PATH`, or you can use rpath to configure the executable to search for dylibs in its parent folder. We've had the least issues with rpath, but YMMV. To configure rpath, you'll need to: