[Bug]: CUDA Error CUDA_ERROR_ASSERT: device-side assert triggered while calling stream_synchronize (cuStreamSynchronize) #1936
Description
Bug Description
That's the situation. I just wanted to try testing the script from the [User Guide] about Parallel Simulation in Ubuntu to test the simulation speed,
Code:
import genesis as gs
import torch
########################## init ##########################
gs.init(backend=gs.gpu)
########################## create a scene ##########################
scene = gs.Scene(
show_viewer = True,
viewer_options = gs.options.ViewerOptions(
camera_pos = (3.5, -1.0, 2.5),
camera_lookat = (0.0, 0.0, 0.5),
camera_fov = 40,
),
rigid_options = gs.options.RigidOptions(
dt = 0.01,
),
)
########################## entities ##########################
plane = scene.add_entity(
gs.morphs.Plane(),
)
franka = scene.add_entity(
gs.morphs.MJCF(file='xml/franka_emika_panda/panda.xml'),
)
########################## build ##########################
# create 20 parallel environments
B = 20
scene.build(n_envs=B, env_spacing=(1.0, 1.0))
# control all the robots
franka.control_dofs_position(
torch.tile(
torch.tensor([0, 0, 0, -1.0, 0, 0, 0, 0.02, 0.02], device=gs.device), (B, 1)
),
)
for i in range(1000):
scene.step()
Unfortunately, an error occurred:
RuntimeError: [cuda_driver.h:operator()@92] CUDA Error CUDA_ERROR_ASSERT: device-side assert triggered while calling stream_synchronize (cuStreamSynchronize)
[Genesis] [01:58:04] [ERROR] RuntimeError: [cuda_driver.h:operator()@92] CUDA Error CUDA_ERROR_ASSERT: device-side assert triggered while calling stream_synchronize (cuStreamSynchronize)
[Genesis] [01:58:04] [INFO] 💤 Exiting Genesis and caching compiled kernels...
[E 11/02/25 01:58:04.913 25015] [cuda_driver.h:operator()@92] CUDA Error CUDA_ERROR_ASSERT: device-side assert triggered while calling stream_synchronize (cuStreamSynchronize)Exception ignored in atexit callback: <function destroy at 0x7f82f90f37e0>
Traceback (most recent call last):
File "/home/gm_funk_zeus/.conda/envs/gs/lib/python3.11/site-packages/genesis/init.py", line 337, in destroy
ti.reset()
File "/home/gm_funk_zeus/.conda/envs/gs/lib/python3.11/site-packages/gstaichi/lang/misc.py", line 199, in reset
impl.reset()
File "/home/gm_funk_zeus/.conda/envs/gs/lib/python3.11/site-packages/gstaichi/lang/impl.py", line 554, in reset
pygstaichi.clear()
File "/home/gm_funk_zeus/.conda/envs/gs/lib/python3.11/site-packages/gstaichi/lang/impl.py", line 532, in clear
self._prog.finalize()
RuntimeError: [cuda_driver.h:operator()@92] CUDA Error CUDA_ERROR_ASSERT: device-side assert triggered while calling stream_synchronize (cuStreamSynchronize)
[E 11/02/25 01:58:05.405 25015] [cuda_driver.h:operator()@92] CUDA Error CUDA_ERROR_ASSERT: device-side assert triggered while calling stream_synchronize (cuStreamSynchronize)
At first, I thought it was my graphics card's performance that was insufficient, and I had to downgrade num_envs to around 600 to run it with an extremely low rate.
But I noticed that your test platform is：RTX 4090 and 14900K.
And the testing platform I use is RTX5880 24G + AMD 96-core processor (RTX5880 24G and RTX 4090 have similar performance)
Therefore, this seems illogical.
[This is my environment information]
Ubuntu22.04 Desktop
It is worth noting that I used Anaconda and followed the PyTorch documentation to perform the installation
pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu126
I'm not sure about this issue because using the exact same operation, it runs normally on my other Windows PC with an RTX 4090 24G + AMD 96-core processor
I don't really think this is a problem with genesis(Because it runs normally under Windows?); it may be related to conda - cuda - driver？
If you have any information about this, thank you very much for being able to share it with me : )
Steps to Reproduce
conda create -n gs python=3.11
conda activate gs
pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu126
torch.cuda.is_available() = True (I checked to ensure CUDA is available)
pip install genesis-world
python main.py
The code is the test code mentioned earlier
Expected Behavior
RuntimeError: [cuda_driver.h:operator()@92] CUDA Error CUDA_ERROR_ASSERT: device-side assert triggered while calling stream_synchronize (cuStreamSynchronize)
[Genesis] [01:58:04] [ERROR] RuntimeError: [cuda_driver.h:operator()@92] CUDA Error CUDA_ERROR_ASSERT: device-side assert triggered while calling stream_synchronize (cuStreamSynchronize)
[Genesis] [01:58:04] [INFO] 💤 Exiting Genesis and caching compiled kernels...
[E 11/02/25 01:58:04.913 25015] [cuda_driver.h:operator()@92] CUDA Error CUDA_ERROR_ASSERT: device-side assert triggered while calling stream_synchronize (cuStreamSynchronize)Exception ignored in atexit callback: <function destroy at 0x7f82f90f37e0>
Traceback (most recent call last):
File "/home/gm_funk_zeus/.conda/envs/gs/lib/python3.11/site-packages/genesis/init.py", line 337, in destroy
ti.reset()
File "/home/gm_funk_zeus/.conda/envs/gs/lib/python3.11/site-packages/gstaichi/lang/misc.py", line 199, in reset
impl.reset()
File "/home/gm_funk_zeus/.conda/envs/gs/lib/python3.11/site-packages/gstaichi/lang/impl.py", line 554, in reset
pygstaichi.clear()
File "/home/gm_funk_zeus/.conda/envs/gs/lib/python3.11/site-packages/gstaichi/lang/impl.py", line 532, in clear
self._prog.finalize()
RuntimeError: [cuda_driver.h:operator()@92] CUDA Error CUDA_ERROR_ASSERT: device-side assert triggered while calling stream_synchronize (cuStreamSynchronize)
[E 11/02/25 01:58:05.405 25015] [cuda_driver.h:operator()@92] CUDA Error CUDA_ERROR_ASSERT: device-side assert triggered while calling stream_synchronize (cuStreamSynchronize)
Screenshots/Videos
No response
Relevant log output
Environment
- OS: Ubuntu 22.04,
- GPU/CPU: RTX 5880 Ada 24G/AMD EPYC 9T24 96-Core Processor
- GPU-driver version : NVIDIA-SMI 570.172.18 Driver Version: 570.172.18
- CUDA / CUDA-toolkit version : CUDA Version: 12.8
Release version or Commit ID
version: 0.3.6
Additional Context
No response