Skip to content

GPU memory leak in _fast_paste_back causes CUBLAS_STATUS_NOT_INITIALIZED crash on long videos #1868

Description

@caiyilian

Description

When processing long videos (>500 frames) with --execution-provider cuda, Deep-Live-Cam crashes partway through with:

  • CUBLAS_STATUS_NOT_INITIALIZED
  • CUDA failure 715: illegal instruction

The crash is caused by a GPU memory leak in _fast_paste_back in modules/processors/frame/face_swapper.py.

Root Cause

_fast_paste_back has a PyTorch CUDA path that creates intermediate tensors on the GPU but never explicitly frees them:

mask_t = torch.from_numpy(mask_crop).float().cuda()
fake_t = torch.from_numpy(bgr_fake_crop).float().cuda()
tgt_t = torch.from_numpy(target_crop).float().cuda()
blended = (mask_t * fake_t + (1.0 - mask_t) * tgt_t).to(torch.uint8).cpu().numpy()

mask_t, fake_t, tgt_t, and blended persist in GPU memory until Python's GC runs. ONNX Runtime (insightface) also holds GPU memory on the same device. Over hundreds of frames, the unreleased PyTorch tensors accumulate, exhausting VRAM and corrupting the CUDA context.

Reproduction

  1. Select a source face image and a target video with ~1000+ frames at 1080p
  2. Run: python run.py -s face.jpg -t test.mp4 -o output.mp4 --execution-provider cuda
  3. Observe crash partway through (typically after ~300-500 frames on a 6GB GPU)

Environment

  • GPU: NVIDIA RTX 2060 6GB
  • Driver: 596.21
  • CUDA: 12.4
  • OS: Windows 11
  • Python: 3.11.14
  • onnxruntime-gpu: 1.18.0
  • PyTorch: 2.6.0+cu124

Fix

  1. Add del mask_t, fake_t, tgt_t, blended after the blend operation to immediately release tensor references
  2. Add periodic gc.collect() + torch.cuda.empty_cache() every ~50 frames in _run_pipe_pipeline
  3. Make release_resources() attempt torch.cuda.empty_cache() via runtime import instead of requiring a module-level HAS_TORCH flag

I have a verified working fix (benchmarked: 1129 frames completed without crash, ~8.9 fps) and can open a PR.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions