Description
When processing long videos (>500 frames) with --execution-provider cuda, Deep-Live-Cam crashes partway through with:
CUBLAS_STATUS_NOT_INITIALIZED
CUDA failure 715: illegal instruction
The crash is caused by a GPU memory leak in _fast_paste_back in modules/processors/frame/face_swapper.py.
Root Cause
_fast_paste_back has a PyTorch CUDA path that creates intermediate tensors on the GPU but never explicitly frees them:
mask_t = torch.from_numpy(mask_crop).float().cuda()
fake_t = torch.from_numpy(bgr_fake_crop).float().cuda()
tgt_t = torch.from_numpy(target_crop).float().cuda()
blended = (mask_t * fake_t + (1.0 - mask_t) * tgt_t).to(torch.uint8).cpu().numpy()
mask_t, fake_t, tgt_t, and blended persist in GPU memory until Python's GC runs. ONNX Runtime (insightface) also holds GPU memory on the same device. Over hundreds of frames, the unreleased PyTorch tensors accumulate, exhausting VRAM and corrupting the CUDA context.
Reproduction
- Select a source face image and a target video with ~1000+ frames at 1080p
- Run:
python run.py -s face.jpg -t test.mp4 -o output.mp4 --execution-provider cuda
- Observe crash partway through (typically after ~300-500 frames on a 6GB GPU)
Environment
- GPU: NVIDIA RTX 2060 6GB
- Driver: 596.21
- CUDA: 12.4
- OS: Windows 11
- Python: 3.11.14
- onnxruntime-gpu: 1.18.0
- PyTorch: 2.6.0+cu124
Fix
- Add
del mask_t, fake_t, tgt_t, blended after the blend operation to immediately release tensor references
- Add periodic
gc.collect() + torch.cuda.empty_cache() every ~50 frames in _run_pipe_pipeline
- Make
release_resources() attempt torch.cuda.empty_cache() via runtime import instead of requiring a module-level HAS_TORCH flag
I have a verified working fix (benchmarked: 1129 frames completed without crash, ~8.9 fps) and can open a PR.
Description
When processing long videos (>500 frames) with
--execution-provider cuda, Deep-Live-Cam crashes partway through with:CUBLAS_STATUS_NOT_INITIALIZEDCUDA failure 715: illegal instructionThe crash is caused by a GPU memory leak in
_fast_paste_backinmodules/processors/frame/face_swapper.py.Root Cause
_fast_paste_backhas a PyTorch CUDA path that creates intermediate tensors on the GPU but never explicitly frees them:mask_t,fake_t,tgt_t, andblendedpersist in GPU memory until Python's GC runs. ONNX Runtime (insightface) also holds GPU memory on the same device. Over hundreds of frames, the unreleased PyTorch tensors accumulate, exhausting VRAM and corrupting the CUDA context.Reproduction
python run.py -s face.jpg -t test.mp4 -o output.mp4 --execution-provider cudaEnvironment
Fix
del mask_t, fake_t, tgt_t, blendedafter the blend operation to immediately release tensor referencesgc.collect()+torch.cuda.empty_cache()every ~50 frames in_run_pipe_pipelinerelease_resources()attempttorch.cuda.empty_cache()via runtime import instead of requiring a module-levelHAS_TORCHflagI have a verified working fix (benchmarked: 1129 frames completed without crash, ~8.9 fps) and can open a PR.