Out of Memory errors with internal resolutions > 1x

[rlyerly]

Hello,

I'm using ParaLLEl-RDP in Retroarch on a system with the following specs:

• Windows 10
• CPU: Intel 8705G with Intel HD 630 iGPU and 8GB RAM
• GPU: Radeon Vega M GL with 4GB HBM (distinct memory from CPU RAM)
• Radeon Adrenalin 20.4.1 drivers, last "official" driver for this GPU
• Retroarch v1.9.0
• Core: Mupen64Plus-Next, 2.1-Vulkan

I can select either GPU via Windows graphics settings. When I use the iGPU, everything works as expected - I'm able to run with internal resolutions above 1x, although framerates drop at 4x because the iGPU is fairly weak. When I use the Vega M GL at 1x internal resolution, it also works correctly. However whenever I set the internal resolution to >1x and launch the emulator, Retroarch opens a window and hangs forever. If I click on the window, Windows says it's not responding and prompts me to force close the emulator.

Here's the Retroarch log for one of these runs, although nothing seems out of the ordinary minus the hang: retroarch-log.txt. I also managed to catch the console log for when it hangs: console-output.txt. It looks like submitting commands the queue for the GPU context fails with out of memory errors (according to the VkResult documentation).

I don't expect ParaLLEl RDP to use 4 GB of memory but maybe I'm wrong. Are there any memory requirements I'm missing?

[rlyerly]

I did some digging and the GPU is definitely not running out of memory, I added print statements and it was only allocating several hundred MBs of GPU RAM. Instead I think my AMD Vulkan driver only allows allocating a puny number of buffers in a descriptor set (maxDescriptorSetStorageBuffersDynamic is 8 according to the Vulkan database). I started seeing the out of memory errors shown in the log above during the 9th call to Renderer::create_buffer (inside Renderer::init_buffers).

It looks like several additional buffers are allocated if ubershaders are not enabled. I tried force-enabling ubershaders using environment variables, but it still hits those errors because Renderer::init_caps() (where ubershaders is initialized via environment variables) is called after Renderer::init_buffers(). However after recompiling with ubershader default initialized to true, I'm able to get in game. Hopefully this is useful!

[Themaister]

parallel-rdp does not use dynamic SSBOs, so that doesn't make much sense. Sounds more like a driver bug. I don't have access to an AMD APU system, so I can't test it if it's APU specific.

There isn't much to go on from the logs, so to get anywhere with this I'd need to know what exactly is failing. A call to vkAllocateMemory perhaps?

[Themaister]

An other user had it working just fine on an AMD APU with 4x and even 8x upscaling (albeit horribly slow as expected), so I don't know.

[rlyerly]

Thanks for replying! Yeah, unfortunately my system is kind of a weird one - it's one of the Kaby Lake G laptops with the Vega M GPU that Intel was supposed to support, then AMD kind of took over support, then neither wanted anything to do with. The driver I have is the last officially supported one, so it looks like the bugs are here to stay.

You're also right about my previous theory being wrong - when I bumped the resolution up to 4x, it throws similar out of memory errors on when creating upscaling_multisampled_hidden_rdram in rdp_renderer.cpp.

If I go back to the original configuration (ubershaders default initialized to false, 2x internal resolution), I see errors starting earlier when creating multisampled_rdram. This is also different from what I observed previously, because this is not the 8th buffer created for the regular GPU heap. I think I'm seeing some variable behavior because of 1) driver bugs and/or 2) when the renderer gets Vulkan errors, it doesn't actually free the GPU memory allocated, meaning my many debugging sessions were eating up GPU resources.

I'm going to try setting up some of the Vulkan validation layers to see if that turns up anything. I'm pretty sure I'm not running out of GPU memory, because I printed the values from get_memory_budget() for all heaps as each allocation was happening and there was plenty of free memory in all of them. There's got to be some other resource that is getting exhausted.