How to configure NVIDIA vGPU on Ubuntu 24.04

• Kernel version: 6.14.0-29-generic
• Graphic card: NVIDIA GTX 1650 Mobile

Configure host

• Download NVIDIA GRID driver version 570.172.07 (or 18.4) from https://alist.homelabproject.cc/foxipan/vGPU/18.4/NVIDIA-GRID-Linux-KVM-570.172.07-570.172.08-573.48/Host_Drivers/NVIDIA-Linux-x86_64-570.172.07-vgpu-kvm.run

• Below are a synthesized instructions from: https://github.com/mbilker/vgpu_unlock-rs, https://github.com/DualCoder/vgpu_unlock & https://gitlab.com/polloloco/vgpu-proxmox. Details about the whys are in there.

• Install prerequisite

  sudo apt install dkms python3 python3-pip git

• Clone repositories

  sudo su
  cd /opt/
  git clone git@github.com:DualCoder/vgpu_unlock.git
  git clone git@github.com:mbilker/vgpu_unlock-rs.git

• Install Rust

  curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

• Install NVIDIA GRID driver

  chmod +x NVIDIA-Linux-x86_64-570.172.07-vgpu-kvm.run
  ./NVIDIA-Linux-x86_64-570.172.07-vgpu-kvm.run --dkms

• Patch the driver to acknowledge consumer graphic card

  sudo sed -i '/#include <linux\/cpuset.h>/a\#include "\/opt\/vgpu_unlock\/vgpu_unlock_hooks.c"' /usr/src/nvidia-570.172.07/nvidia/os-interface.c
  echo 'dflags-y += -T /opt/vgpu_unlock/kern.ld' | sudo tee -a /usr/src/nvidia-570.172.07/nvidia/nvidia.Kbuild

• Reinstall the driver

  dkms remove -m nvidia -v 570.172.07 --all
  dkms install -m nvidia -v 570.172.07

• 2nd patch

  `cd /opt/vgpu_unlock-rs`
  `cargo build --release`
  `mkdir /etc/systemd/system/nvidia-vgpud.service.d`
  `mkdir /etc/systemd/system/nvidia-vgpu-mgr.service.d`
  `echo -e "[Service]\nEnvironment=LD_PRELOAD=/opt/vgpu_unlock-rs/target/release/libvgpu_unlock_rs.so" > /etc/systemd/system/nvidia-vgpud.service.d/vgpu_unlock.conf`
  `echo -e "[Service]\nEnvironment=LD_PRELOAD=/opt/vgpu_unlock-rs/target/release/libvgpu_unlock_rs.so" > /etc/systemd/system/nvidia-vgpu-mgr.service.d/vgpu_unlock.conf`

• vGPU profiles

  mkdir /etc/vgpu_unlock
  vim /etc/vgpu_unlock/profile_override.toml

  • add the following into it:

  # override for a specific profile, of which only one can be used
  [profile.nvidia-256]
  num_displays = 1
  display_width = 1920
  display_height = 1080
  max_pixels = 2073600
  cuda_enabled = 1
  frl_enabled = 1
  vgpu_type = "NVS"
  framebuffer = 0x74000000
  framebuffer_reservation = 0xC000000
  
  # config for specific a mdev which will be attached to a specific VM.
  [mdev.16c517a1-3724-47a0-b8fc-7bfd0d110fb6]
  framebuffer = 0x1A000000
  framebuffer_reservation = 0x6000000

• Reboot

• Verify

  nvidia-smi vgpu

  • Should show a table of information.

• List mdev types / vGPU profiles

  mdevctl types

  • Choose one a Q type for gaming.

• Create an mdev device

  • Where "nvidia-256" is the profile type.

  echo $(uuidgen) | sudo tee /sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/nvidia-256/create

• Add fixes for libvirt/qemu

  • Hooks to fix vram assignment and auto re-create mdev on startup. Those files are from this repository.

  cp qemu /etc/libvirt/hooks/qemu
  cp profile_override.toml /etc/vgpu_unlock/profile_override.toml

  • Modify the params to your own need.

• Attach it to the VM in virt-manager

• Install license server

  • Follow instructions from: https://git.collinwebdesigns.de/oscar.krause/fastapi-dls

Limitations

• There is no dynamic ram assignment as the 2 values may suggset (framebuffer and framebuffer_reservation)
• The profile override to assign specific profile per mdev or VM is broken. So a workaround is to use qemu hooks to override the file /etc/vgpu_unlock/profile_override.toml, upon VM preparation and restore is upon startup for each VM.

Configure guest

Windows 10 guest

• Download & install guest driver
• https://alist.homelabproject.cc/d/foxipan/vGPU/18.4/NVIDIA-GRID-Linux-KVM-570.172.07-570.172.08-573.48/Guest_Drivers/573.48_grid_win10_win11_server2022_dch_64bit_international.exe
• Follow those instructions to patch the license: https://git.collinwebdesigns.de/vgpu/gridd-unlock-patcher/

Ubuntu 24.04 guest

• Install prerequisite

  sudo install pkg-config libglvnd-dev

• Download driver

  • https://alist.homelabproject.cc/d/foxipan/vGPU/18.4/NVIDIA-GRID-Linux-KVM-570.172.07-570.172.08-573.48/Guest_Drivers/NVIDIA-Linux-x86_64-570.172.08-grid.run

• Install driver

  chmod +x NVIDIA-Linux-x86_64-570.172.08-grid.run
  sudo ./NVIDIA-Linux-x86_64-570.172.08-grid.run --dkms

• Follow those instructions to patch the license: https://git.collinwebdesigns.de/vgpu/gridd-unlock-patcher/

Display

• Virt-manager can't directly use the output of the vGPU so we must use a streaming app.

• https://looking-glass.io, uses a shared memory between the guest and host. (Recommended)

• Sunshine + Moonlight:

  • Server to be installed on the guest: https://docs.lizardbyte.dev/projects/sunshine/latest/md_docs_2getting__started.html
  • Client to be installed on the host: https://github.com/moonlight-stream/moonlight-qt

Disclaimer

This guide is provided for informational purposes only. The author makes no representations or warranties of any kind, express or implied, about the completeness, accuracy, reliability, suitability, or availability with respect to the information, products, services, or related graphics contained in this guide for any purpose. Any reliance you place on such information is therefore strictly at your own risk.

In no event will the author be liable for any loss or damage including, without limitation, indirect or consequential loss or damage, or any loss or damage whatsoever arising from loss of data or profits arising out of, or in connection with, the use of this guide.

The procedures described herein involve modifications to your system that could potentially lead to instability, data loss, or hardware damage. Proceed with caution and ensure you have adequate backups of your data before attempting any of these steps. The author is not responsible for any adverse outcomes.

License

GNU General Public License v3.0