High frequency Tex.FromColors() and Sprite.FromTex() causes HoloLens app to crash

[llfab]

I want to show video frames within the HoloLens app. Therefore, I am sending accross http messages that contain image/jpeg with 640x480. After receiving them in the HL app, I transfer each frame into a Sprite using Tex.FromColors() followed by Sprite.FromTex(). Since I am trotteling the video frames from the sender side to 10 fps that is displayed nicely and with not such a bad latency. However, after some time it seems the incoming video frames cause the app to crash. This does not happen in the emulator.

As far as I understand, Tex.FromColors / Sprite.FromTex reserves memory on the GPU, hence, it should not be called with high frequency, right? Any way to boost this for high frequency images?

In 2D UI's you typically use WriteableBitmap for such a case, which re-uses the memory and just overrides the color information for each pixel. Any way to do this with SK?

Also I do not see any Dispose() functions for the more heavy weight elements such as Tex. How do you propose to release them?

[maluoi]

Tex.FromColors creates a whole new texture object, and sets it up as an immutable graphics resource. Which is a little slower to create, and definitely isn't friendly towards frequent creation, but theoretically sits in a faster chunk of GPU memory. The best way to do what you're trying to do is to re-use a texture+sprite that you already have! If you're using SpriteType.Single, then it does maintain a direct reference to the Tex asset that's provided, so you won't need to create a new Sprite asset either.

You'll want to create a new Tex(TexType.ImageNoMips | TexType.Dynamic), and then update its contents each time with Tex.SetColors. If you update an image with Tex.SetColors more than once, SK will automatically recreate the texture with the Dynamic flag anyhow. This sets the texture up in GPU memory that's much more friendly to frequent updates :)

[llfab]

Thanks. I have implemented accordingly and it seems to work nicely. For other images (1024x1024) that get updated only once in a while I have done the same an measured the timing. It seems that the SetColors takes about 150 msec on the Hololens 2. I am using Color32[], fill the array accordingly and then use void SetColors(int width, int height, Color32[]& data). Since, the color info comes from our own image container (managed data), it seems this cannot be optimized right?

[maluoi]

Is that with mip-map generation on? Mip-maps are currently built CPU side, it's usually quite fast, but I haven't profiled that on HL2. I'm not certain which would be slower, mip generation or static texture creation, but I'll definitely take a peek. That does sound a bit excessive.

One thing of interest here is that next major update will also include async tools for texture assets! So there'll be a number of options for non-blocking image parsing, mips, and color uploads.