Memory pool for RenderTargets to help with complex EffectComposer pipelines

[bhouston]

Description of the problem

I have a lot of effects on my ThreeJS scenes. Right now with the way ThreeJS's effects are designed, they each allocate their own render targets, even their intermediate/temporary render targets. This leads to somewhere around 50 render targets allocated for my specific use case. But really only a couple are ever used at the same time, the rest are just setting there wasting memory and causing out of memory errors are various devices, especially high resolution (QHD/4K) + low memory mobile devices.

I am thinking that if we create a memory pool ( https://en.wikipedia.org/wiki/Memory_pool ) for RenderTargets we could decrease the maximum amount of memory required for complex EffectComposer pipelines without sacrificing speed or quality.

The design would be something like:

RenderTargetPool:
allocate( width, height, options ) -> renderTarget;
free( renderTarget )
nextFrame() - increase current frame.
disposeOld( oldness ) - to dispose all targets not recently used (could be all)
disposeAll() -- to be called once done or in the case of a resize (see below.)

This memory pool would track which render targets are currently in use (which haven't been returned) and allocate new ones if needed. But it would try to recycle as much as possible. I'd keep it simple and track the current frame and allow explicit disposal of those render targets that haven't been used in a recent frame (specified by the user - but I think a value of 10 is pretty reasonable for most use cases) or all current render targets (if ending the rendering.)

This will automatically minimally allocate render targets.

I guess the only issue is how to handle resizes. Right now ThreeJS will resize render targets when the render region changes shape. I wonder if we adopt the RenderTargetPool we could just dispose all in that case and recreate render targets of the new size -- I think that is how it works underneath anyhow.

Only intermediate render targets can be used via the pool. Those that persist such those that back up textures attached to objects need to have explicitly controlled lifetimes.

Thoughts welcome.... this is just an idea on how to solve our issue while making Three.JS easier and more efficient.

Three.js version

• Dev
• r84

Browser

• All of them

OS

• All of them

[AUTOMAKHAN]

Memory pooling could be extended beyond just RenderTargets. There is a lot of GC churn each render frame due to simple temporary objects like Vector3, etc.

Object pools might make the code ugly, and can introduce some nasty silent bugs if not careful. It might be worth the performance cost to keep the code clean. But it would definitely be a huge performance boost and cut down on the GC freezes. I'm on the fence about it.

[mrdoob]

There is a lot of GC churn each render frame due to simple temporary objects like Vector3, etc.

We should really clean those up...

[wrr]

RenderTargetPool would be a very useful addition. I wonder if a lower level and more explicit interface wouldn't be easier to reason about. For example with 3 operations:

1. I need a render target (width, height, options)
2. I no longer use the render target (so it can be reused) but I will need a target with the same width, height, options again.
3. I no longer use the render target and won't need such target again.

[twerdster]

Just thought it might be worth pointing out here that GPUImage by Brad Lawson uses a framebuffer cache (lines 69-125) specifically for this problem. An similar approach is the approach by Metal image API which can have a framebuffer defined as a temporary buffer (MPSTemporaryImage) and once it is used and noone else depends on it, it is recycled.

[DavidPeicho]

I think this is a much needed improvement. I am working on some SSS shader using MRTs. However, passes like the SSAO or SSR are simply creating their own GBuffer-like. A user that would then use something like SSS -> SSR -> SSAO would end up with a crazy amount of render targets. I actually never use the composer, but I wanted something people can just drop in.

Related: #15694