Unable to run examples on Jetson AGX Orin due to libnvdla_compiler.so not found

[BradySheehan]

Hi,
I'm having issues running the default examples provided in the package.

I have a Jetson AGX Orin I configured using the Jetpack SDK Manager that has the following configuration

$ jetson_release 
Software part of jetson-stats 4.2.12 - (c) 2024, Raffaello Bonghi
Model: NVIDIA Jetson AGX Orin Developer Kit - Jetpack 6.0 [L4T 36.3.0]
NV Power Mode[2]: MODE_30W
Serial Number: [XXX Show with: jetson_release -s XXX]
Hardware:
 - P-Number: p3701-0005
 - Module: NVIDIA Jetson AGX Orin (64GB ram)
Platform:
 - Distribution: Ubuntu 22.04 Jammy Jellyfish
 - Release: 5.15.136-tegra
jtop:
 - Version: 4.2.12
 - Service: Active
Libraries:
 - CUDA: 12.2.140
 - cuDNN: 8.9.4.25
 - TensorRT: 8.6.2.3
 - VPI: Not installed
 - Vulkan: 1.3.204
 - OpenCV: 4.5.4 - with CUDA: NO

I then started a container with ./docker/run.sh after cloning the repo, but when launching any of the examples, I get the following error:

jetson-inference$ ./docker/run.sh 
ARCH:  aarch64
reading L4T version from /etc/nv_tegra_release
L4T BSP Version:  L4T R36.3.0
CONTAINER_IMAGE:  dustynv/jetson-inference:r36.3.0
DATA_VOLUME:      --volume /home/bsheehan/Development/jetson-inference/data:/jetson-inference/data --volume /home/bsheehan/Development/jetson-inference/python/training/classification/data:/jetson-inference/python/training/classification/data --volume /home/bsheehan/Development/jetson-inference/python/training/classification/models:/jetson-inference/python/training/classification/models --volume /home/bsheehan/Development/jetson-inference/python/training/detection/ssd/data:/jetson-inference/python/training/detection/ssd/data --volume /home/bsheehan/Development/jetson-inference/python/training/detection/ssd/models:/jetson-inference/python/training/detection/ssd/models --volume /home/bsheehan/Development/jetson-inference/python/www/recognizer/data:/jetson-inference/python/www/recognizer/data 
root@ubuntu:/opt/jetson-inference/build/aarch64/bin# ./detectnet images/peds_0.jpg images/test/peds_0.jpg
./detectnet: error while loading shared libraries: libnvdla_compiler.so: cannot open shared object file: No such file or directory

Notably, the file is available at /usr/lib/aarch64-linux-gnu/nvidia/libnvdla_compiler.so which I believe is reasonable, since it should get mounted with the --runtime=nvidia argument for docker.

My LD_LIBRAY_PATH by default is /usr/local/cuda/compat:/usr/local/cuda/lib64: and I tried adding /usr/lib where the .so is to the LD_LIBRARY_PATH, but it did not change my error.

I tried rebuilding the source inside the container by following the instructions here - https://github.com/dusty-nv/jetson-inference/blob/master/docs/building-repo.md , but I encounter a different error when running make -j$(nproc) related to a bunch of undefined references to nvdla full log attached here build_error_output.txt .

I noticed that it was recommended to use the jetson-containers build jetson-inference command for building docker containers on the jetson for use with jetson-inference but that also gives an error during step6 of the build,

Step 6/6 : RUN cd /tmp/opencv && ./install.sh || ./build.sh || echo "BUILD FAILED (OpenCV ${OPENCV_VERSION})"
 ---> Using cache
 ---> bce5c916341b
Successfully built bce5c916341b
Successfully tagged jetson-inference:r36.3.0-opencv
-- Testing container jetson-inference:r36.3.0-opencv (opencv:4.8.1/test.py)

docker run -t --rm --runtime=nvidia --network=host \
--volume /home/bsheehan/jetson-inference/jetson-containers/packages/opencv:/test \
--volume /home/bsheehan/jetson-inference/jetson-containers/data:/data \
--workdir /test \
jetson-inference:r36.3.0-opencv \
/bin/bash -c 'python3 test.py' \
2>&1 | tee /home/bsheehan/jetson-inference/jetson-containers/logs/20241219_162719/test/jetson-inference_r36.3.0-opencv_test.py.txt; exit ${PIPESTATUS[0]}

testing OpenCV...
Traceback (most recent call last):
  File "/test/test.py", line 4, in <module>
    import cv2
ModuleNotFoundError: No module named 'cv2'
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/bsheehan/jetson-inference/jetson-containers/jetson_containers/build.py", line 112, in <module>
    build_container(args.name, args.packages, args.base, args.build_flags, args.build_args, args.simulate, args.skip_tests, args.test_only, args.push, args.no_github_api, args.skip_packages)
  File "/home/bsheehan/jetson-inference/jetson-containers/jetson_containers/container.py", line 154, in build_container
    test_container(container_name, pkg, simulate)
  File "/home/bsheehan/jetson-inference/jetson-containers/jetson_containers/container.py", line 327, in test_container
    status = subprocess.run(cmd.replace(_NEWLINE_, ' '), executable='/bin/bash', shell=True, check=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bsheehan/miniconda3/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'docker run -t --rm --runtime=nvidia --network=host --volume /home/bsheehan/jetson-inference/jetson-containers/packages/opencv:/test --volume /home/bsheehan/jetson-inference/jetson-containers/data:/data --workdir /test jetson-inference:r36.3.0-opencv /bin/bash -c 'python3 test.py' 2>&1 | tee /home/bsheehan/jetson-inference/jetson-containers/logs/20241219_162719/test/jetson-inference_r36.3.0-opencv_test.py.txt; exit ${PIPESTATUS[0]}' returned non-zero exit status 1.

if I use the --skip-tests=all argument, I get further and fail with the error here jetson_containers_build_error.txt related to #include <gst/webrtc/webrtc.h> with fatal error: gst/webrtc/webrtc.h: No such file or directory.

I had a suspicion that the problems may be related to my tensorrt installation, but I have not been able to conclude that concretely. It looks like my nvidia-tensorrt version is 6.0+b106 and the supported packages like libnvinfer8 are version 8.6.2.3-1+cuda12.2 but I think that makes sense.

Any ideas what else I can try? Am I following the wrong instructions?

Thank you for the help,
Brady

[tyoh-love]

i got same error!!!