[🐛 bug report] Rendering with cuda_* variants fails even for trivial scenes

[mworchel]

Summary

Rendering with the cuda_* variants fails even the most trivial scenes fails, if they contain geometry:

Dr.Jit encountered an unrecoverable error and will now shut
down. Please re-run your program in debug mode to check for
out-of-bounds reads, writes, and other sources of undefined
behavior. You can do so by calling

   dr.set_flag(dr.JitFlag.Debug, True)

at the beginning of the program. If these additional checks
fail to pinpoint the problem, then you have likely found a
bug. We are happy to help investigate and fix the problem if
you can you create a self-contained reproducer and submit it
at https://github.com/mitsuba-renderer/drjit.

The error message of this specific failure is as follows:
>>> cuda_check(): API error 0718 (CUDA_ERROR_INVALID_PC): "invalid program counter" in [D:\a\drjit\drjit\ext\drjit-core\src\init.cpp:462](file:///D:/a/drjit/drjit/ext/drjit-core/src/init.cpp:462).

System configuration

System information:

OS: Windows-10
CPU: Intel64 Family 6 Model 183 Stepping 1, GenuineIntel
GPU: NVIDIA GeForce RTX 3090
Python: 3.12.8 | packaged by conda-forge | (main, Dec 5 2024, 14:06:27) [MSC v.1942 64 bit (AMD64)]
NVidia driver: 566.36
CUDA: 12.4.99
LLVM: -1.-1.-1

Dr.Jit: 1.0.1
Mitsuba: 3.6.0
Is custom build? False
Compiled with: MSVC 19.42.34433.0
Variants:
scalar_rgb
scalar_spectral
scalar_spectral_polarized
llvm_ad_rgb
llvm_ad_mono
llvm_ad_mono_polarized
llvm_ad_spectral
llvm_ad_spectral_polarized
cuda_ad_rgb
cuda_ad_mono
cuda_ad_mono_polarized
cuda_ad_spectral
cuda_ad_spectral_polarized

Description

On my system even trivial scenes fail to render if they contain geometry. Here is a minimal reproducer:

import mitsuba as mi
import drjit as dr

mi.set_variant('scalar_rgb')
dr.set_flag(dr.JitFlag.Debug, True)

scene_dict = {
    'type': 'scene',
    'integrator': {
        'type': 'path',
    },
    'sensor': {
        'type': 'perspective',
        'to_world': mi.ScalarTransform4f().look_at(
            origin=[0, 0, 5],
            target=[0, 0, 0],
            up=[0, 1, 0]
        ),
    },
    # Without 'shape', rendering succeeds (but of course the output is 0)
    'shape': {
        'type': 'sphere',
        'emitter': {
            'type': 'area',
            'radiance': {'type': 'rgb', 'value': [10, 10, 10], } 
        }
    }
}

scene  = mi.load_dict(scene_dict)
mi.Bitmap(mi.render(scene, spp=64))

The reported error message varies between CUDA_ERROR_INVALID_PC and CUDA_ERROR_ILLEGAL_ADDRESS (if I comment out the sensor to_world transform, then the latter is reported). With the scalar_rgb variant, the scene is rendered.

I cannot pinpoint exactly when this started happening and I used Mitsuba 3 previously on this system. I've tested different mitsuba versions (3.4.1, 3.5.2, 3.6.0) but it fails for all of them. The driver is up-to-date (I thought that updating the driver might fix it, but it didn't). It might still be a driver issue?

Let me know if I can do anything to help the investigation (e.g. running with certain flags or compiling/testing a different branch).

[wjakob]

It is an NVIDIA driver code generation bug that they will fix in the upcoming R570 driver: mitsuba-renderer/drjit#296.
Until then, the only thing you can do is to downgrade your driver to an older version.

[mworchel]

Thank you so much, downgrading the driver fixed it! Sorry for opening the issue here, I should've looked through the DrJit issues first.