[Research] Graphics APIs (OpenGL/Vulkan), GPU drivers, video acceleration, etc.

[kylewlacy]

We'll need to figure out how to handle packages that use GPUs in some fashion or another, e.g. through graphics APIs like OpenGL or Vulkan, hardware-accelerated video encoding/decoding APIs like VA-API / NVENC, GPGPU APIs like CUDA/ROCm/OpenCL, and so on.

Of course, this is most important for packages at runtime, but it'd be good to understand if / how this could fit in at build time too. (Seems kinda niche, but I don't see why we shouldn't allow a (probably sandboxed) build from running a program that uses, say, CUDA).

At a high level, graphics APIs are "just" shared libraries. But I don't think we can get away with treating them like most other shared libraries. One of the most obvious points of friction is that Nvidia's official graphics driver isn't part of Mesa, unlike the drivers from Intel, AMD, and other graphics driver vendors. And realistically, even if we could just ship Mesa as-is, that would lock each installed package to a specific Mesa version, which would mean it would (probably?) need to be rebuilt e.g. to support newer hardware, or even just to get driver fixes.

So, we'll need to figure out how to handle graphics drivers in Brioche packages.

[kylewlacy]

Some notes from other ecosystems:

Nix

From what I've gleaned, NixOS puts the library for your hardware drivers under /run/opengl-driver/lib (for OpenGL at least). Then, in Nixpkgs, packages look for /run/opengl-driver/lib by default, although this can also be adjusted using $LD_LIBRARY_PATH (n.b. I'm not sure if this path is actually used explicitly in Nixpkgs, or if it's just that NixOS includes it in $LD_LIBRARY_PATH by default?)

It seems like for non-NixOS systems, Nixpkgs doesn't really do anything to help out-of-the-box, so some separate tools have cropped up to help with this, like nixGL.

Further reading:

• https://nixos.wiki/wiki/OpenGL
• https://nixos.wiki/wiki/Nixpkgs_with_OpenGL_on_non-NixOS
• libGL not working on non-NixOS (without setting up) NixOS/nixpkgs#9415
• https://github.com/nix-community/nixGL

Flatpak

Flatpak has the idea of "extensions", which I think are basically separate packages you can layer on top of a package. Then, they package the org.freedesktop.Platform.GL.nvidia pack, which redistributes Nvidia's drivers as an extension, I believe. That means you do need to install this pack to get hardware acceleration with Nvidia, but it can be updated separately from the packages that use it.

Further reading:

• https://docs.flatpak.org/en/latest/extension.html
• https://github.com/flathub/org.freedesktop.Platform.GL.nvidia
• https://blogs.igalia.com/vjaquez/digging-further-into-flatpak-with-nvidia/

[nz366]

Maybe some examples of using system libraries would show the whole picture.