Optimise saved texture usage for properies such as box-shadow

[ZilverBlade]

I've been optimising how i create RmlUi textures by loading them into an atlas. This has been marvelous for performance, as it caches styles that aren't changing, and when recreating it's fast. However i've noticed that despite differing elements being visually identical, rmlui saves each image seperately?

This is a view of my texture atlas at some point in time.

Since i have many elements that have a box-shadow effect, rmlui seems to save each one seperately. Moreover, since the box-shadow does not save anything about the backdrop-filter or anything (so no baked in blur or anything), i dont see why it needs to recreate the texture for each element?

I propose that maybe for each element, compute some sort of hash based on the properties whenever they change, so that it can reuse other elements with the same properties whenever possible, reducing memory usage and resource creation, however I'm not sure if this could break other important behaviour.

[mikke89]

Hey. The atlas definitely looks like an interesting optimization technique. It's been on my mind for some time to do something like this for font textures, but not to include other generated textures. Could be interesting to consider for the library. Although these techniques may have some drawbacks too, particularly when combined with transforms, that your textures can end up sampling for neighboring sprites in the sprite sheet due to filtering. Maybe it's better left as a user-optimization technique.

Anyway, to answer your question. And as is the usual answer with many questions like these, it's mainly a matter of spending our limited resources on the things that matter, on the needs we can identify. And this is one need that I haven't seen before, I am definitely happy to see that you're finding the feature useful and applying it widely! I can see how it might be used a lot in some styles.

With any optimization proposal, I think we should identify the potential gains. In particular, how many of the textures in your case can actually be reused? And how much memory and generation time could we potentially save? Would be nice with some real-world numbers.

I definitely agree that in many of these cases there is no need to produce unique textures, and would support adding some sort of cache for box shadows if the numbers look good. And might I add - while we're at it - perhaps we could include border-backgrounds too more generally? Decorators and filters might be more complicated, so I would stay away from them at least in the first iteration.

I think it's a bit more complicated then you make it out to be though. There are several per-element factors that will cause them to be unique, despite the box-shadow properties being the same. To mention some: opacity, laid out width and height, border-radius.

Also, I wouldn't focus so much on the hashing part, I would consider that more of an optimization. First we need to identify all the unique factors and make sure they match whatever is in the cache. A hash table would make sense but we should also make sure they are actually unique - i.e. handle hash collisions.

Let me know if I can help out in any way, or if I can provide some pointers to start.

[ZilverBlade]

Alright, to answer your question

In particular, how many of the textures in your case can actually be reused?
Right now since my UI is work in progress, there are probably less textures than i will be using in the end product.

Here is a screen shot of it being used in the UI (not everything is visible, there are quite a few more elements), i've highlighted the main elements that are used in the UI several times. And as you can see in the atlas. The only thing that isn't actually in the atlas is the big panel, which is actually what led me to opening this issue. Since i have multiple panels that open up when you press a button, a new box shadow is created for each one of them, and since those panels are quite huge, it takes up a lot of space in my atlas, this is what that looked like

It looks like a lot of space was wasted, but if you look closely, you can see some outlines (from those panels), and this ended up overfilling my 4096x4096 atlas, so i switched to using `filter: drop-shadow()` for that instead, in the picture i opened the issue with. So this optimisation would not only reduce memory usage significantly in certain cases (imagine, a panel with a drop-shadow that's the almost size of the screen that's being rendered, instead of just a portion; especially on 4K displays, i was just using a 1080p display!). So a lot of textures would be reused. Now i must also add, in my UI i have a lot of moving objects, so i expect recalculations, and it's nice that RmlUi immediately releases textures that are not needed, so the atlas doesnt have "dead" pages.

In short, since most UI elements (such as buttons, drop-downs, text inputs, etc.) will be visually identical to one another, i think that orders of magnitude of time and space could be saved (think of a setting menu, which has a lot of UI controls).
Anyhow, for now, I don't think I will be running into running out of space in my atlas any time soon anymore, however it's still a concern i have (e.g. having font effects on languages such as chinese/japanese which can have over hundreds or even a thousand unique characters which actually need the space, will not have it)

[mikke89]

Thanks! Yeah, that panel one is especially motivating. Although, I guess normally one would only have one screen-sized panel like that at a time? But I'm not sure how good we are at destroying them, don't think we do that immediately just by hiding them. Mainly when the geometry changes or the element is destroyed, I believe. That could be a thing to look at as well, at least for large elements.

Generally, I am positive about implementing a cache for box shadows. It's mainly a matter development resources I would say, but a PR would very much be welcome.

[ZilverBlade]

Okay then, I'd love to give it a shot, but I'm busy myself, and I don't exactly know the structure of the RmlUi project. If I get around to figuring it out, I might make a PR out of this issue

[ZilverBlade]

I managed to find some time to toy around with the rmlui box shadows, and i've come across some interesting design choices.

• 1 I've noticed that the texture callback for some reason requires the original element to be drawn? I understand the list of box shadows, that can be extraced, however, in GeometryBoxShadow.cpp, background_border_geometry.Render(element_offset_in_texture); is called? What does this exactly do? I don't suppose it actually draws the box shadow on top of the background_border_geometry, which comes from the element that has the box-shadow property.
• 2 CallbackTexture is non-copyable, but it has a Release() function. Would it perhaps be a better idea to ref count it manually (or by use of a shared pointer?) like this, we can safely share textures between resources that require it.

Thanks

[mikke89]

Thanks for looking into this, very much appreciated! I'll have a look at it as soon as I have some time. To address your questions:

1. If I remember correctly, we include the background and border into the same box shadow texture. When rendering, we only draw this box shadow instead of the background and border. So really the texture is background+border+box-shadow. The purpose is to save a draw call when both box shadow and border-backgrounds are present. Maybe there were some other reasons too like how it appears under transformations, I don't remember exactly. I understand the confusion though, this might have been named or documented better.
2. The Release() function releases the underlying resource from the render interface. As opposed to "resetting" the pointer in the unique pointer terminology. The resource can still be re-generated as needed from the same object. Do we actually need shared ownership? Establishing clear ownership rules with unique ownership is almost always preferred, and I haven't seen the need for shared ownership so far.

[ZilverBlade]

Okay, for the first bullet point, if it's not viable to draw the background again after having precomputed the drop shadow, is there maybe a better way to copy the geometry information to be able to cache it? I do think that making the background pass of the geometry seperate would ease up the process though, especially considering that if you have multiple elements with different colours for instance, each shadow would need to be re-rendered because of the background.
As for the second point, the reason I suggested a shared pointer, is that if an element changes (for instance, the transform changes), this means that it will need a new callback texture. However, we don't want to destroy the existing texture, in the event that another element is using that callback texture already. This would prevent stuff like use after free, or memory leaks. Maybe it's a design problem on my end, but this is how I envisioned it.

[mikke89]

I do think that making the background pass of the geometry seperate would ease up the process though,

Hm, yeah, there are different trade-offs here depending on use cases. My gut tells me that if we use the same box-shadow on different elements, they are somewhat likely to use the same background-border too? The alternative is to introduce additional draw calls for everyone that uses backgrounds/border+box-shadow. But we are making an optimization decision on very little data here. It seems a bit safer to go with the current approach of combined textures, so that not all users pay for this potential optimization.

As for the second point, the reason I suggested a shared pointer, is that if an element changes (for instance, the transform changes), this means that it will need a new callback texture

The way I see it - we introduce a box-shadow cache. When the element's properties change, we need to lookup the box shadow from the cache. If there is no matching one there, we create a new (unique) one. The cache entry owns the unique texture, and hands out weak references to that cache entry to elements. Thus, the texture is still unique, but the cache entry itself is shared, but still managed by the cache itself. This can either be modeled with manual reference counting in the cache implementation, or using shared and weak pointers.

We do a similar thing for the SVG cache in particular. However, that one is more complicated as it has multiple layers of cache.

[ZilverBlade]

Oh I see, then that solves the approach for sharing the texture. By the way, I guess it'd true that making that decision on seperating the draw call could penalise some users in performance, however this then becomes a tradeoff between memory, and draw call usage. To be fair, I suppose that people indeed will reuse the same Background/border for the texture, so it seems more of a problem on dealing with the passing of information. Perhaps we can just pass the border and the colour information, and then draw a normal mesh instead of drawing the element? Or do we also draw the texture of the element?

[mikke89]

To be fair, I suppose that people indeed will reuse the same Background/border for the texture, so it seems more of a problem on dealing with the passing of information

Right, we should be able to pass that information to the cache and make it part of the cache key, I don't think it should introduce much more complexity.

Perhaps we can just pass the border and the colour information, and then draw a normal mesh instead of drawing the element? Or do we also draw the texture of the element?

I am not sure I fully follow here, I think the cache should store the texture, and I guess it makes sense for the geometry associated for that texture too. At least I think that should be equivalent for every user of that cache entry?

[ZilverBlade]

What I meant in my question is if the box shadow caches the entire element, possibly caching a decorator too (like a sprite)? Or is it purely the vertex information (so the colour, the border, etc.). If it doesn't, then I should be able to find a way to precompute a mesh from the element, and then move that into the texture callback, likewise with the box shadow list. (EDIT: i just looked again at the function for drawing the geometry, it seems as if the texture is null when calling background_border_geometry.Render(element_offset_in_texture); so ignore what i said). By the way, how do I make a weak reference to the callback texture? There any usage of weak references in the project, or do we need to implement it? We also need to ref count it internally to make sure we destroy when nothing else is using it

[mikke89]

Decorators can't really be included in this cache, since those can do completely different things with their own geometry, shaders, or have animations.

You can use the Texture class which is a view of the callback texture (no reference counting, so you need to handle lifetimes). For reference counting, you can either do it manually inside the cache, or use SharedPtr.

[ZilverBlade]

Okay, I think the latest commit I've pushed is correct, could you review the code when you have time to see if it's in order? I haven't been able to test it because I couldn't figure out how to build the samples with glfw, so I could only run win32 opengl2 which had no filter support, although in the debugger everything seems to work as expected, textures are being reused, stuff is being freed when it's no longer needed, etc.