Improve zoomed debug hitboxes performance

[MaybeMaru]

This PR clamps the rendered hitbox just outside the bounds of the camera. This has to be done since, by the way openfl works, the rects get cached as bitmaps and when zoomed can easily generate absurdly large bitmaps, which cause the framedrops.
It also fixes the hitbox being slightly round around the edges.

This PR has been tested on both native and web targets and seem to perform with the same consistent framerate.
This solution is only necessary for non-flash targets, as flash doesn't have this issue.

It is important to note, however, that this fix is only temporal. Even though it fixes the issue, a shader solution (such as the one proposed in #3168) would be the way to go to completely skip all the bitmap caching openfl has to handle in the background.

Before	After

[MaybeMaru]

Another change that I've been thinking about is keeping the 1 pixel thickness consistent between camera zooms, I can push that too if its a feature that could be wanted here.

[Geokureli]

I see no immediate reasons not to merge this, but I have questions:

1. Can you share your testing environment?
2. I like the MITER, does that improve the performance or was it purely aesthetic? I imagine rounded corners are harder to draw, and cheems solution also avoided rounded corners, which may have contributed to those gains

... the rects get cached as bitmaps...

3. This doesn't make any sense to me, I don't see why they would be cached as a bitmap, can you back up this claim?

Even though it fixes the issue, a shader solution (such as the one proposed in #3168)

4. At best this alleviates specific cases of bad debug draw performance, there are many that have nothing to do with zoom or off-screen objects

It might be a minute before I can test this thoroughly, but it seems promising

[MaybeMaru]

1. Can you share your testing environment?

Here it is, it's just a basic FlxState with a default flixel sprite and some code to move the camera around to zoom in on the hitbox.

package;

import flixel.*;
import flixel.util.*;

class PlayState extends FlxState
{
	override public function create()
	{
		super.create();

		FlxG.fixedTimestep = false;
		FlxG.debugger.drawDebug = true;
		FlxG.camera.bgColor = FlxColor.GRAY;

		var flixel = new FlxSprite();
		@:privateAccess flixel.checkEmptyFrame();
		flixel.scale.set(20, 20);
		flixel.updateHitbox();
		add(flixel);
	}

	override public function update(elapsed:Float)
	{
		super.update(elapsed);

		if (FlxG.keys.pressed.E)
			FlxG.camera.zoom += 2 * FlxG.camera.zoom * elapsed;
		if (FlxG.keys.pressed.Q)
			FlxG.camera.zoom -= 2 * FlxG.camera.zoom * elapsed;

		if (FlxG.keys.pressed.W)
			camera.scroll.y -= 200 * elapsed / FlxG.camera.zoom;
		if (FlxG.keys.pressed.S)
			camera.scroll.y += 200 * elapsed / FlxG.camera.zoom;
		if (FlxG.keys.pressed.A)
			camera.scroll.x -= 200 * elapsed / FlxG.camera.zoom;
		if (FlxG.keys.pressed.D)
			camera.scroll.x += 200 * elapsed / FlxG.camera.zoom;
	}
}

2. I like the MITER, does that improve the performance or was it purely aesthetic? I imagine rounded corners are harder to draw, and cheems solution also avoided rounded corners, which may have contributed to those gains

I applied the MITER value purely for aesthetic reasons to give hitboxes a straighter look. In my testing, they didn’t seem to create any performance difference as opposed to keeping the rounded edges. Though now that you mention it, they may perform better. I'll have to get back to you on that.

3. This doesn't make any sense to me, I don't see why they would be cached as a bitmap, can you back up this claim?

In OpenFL, it’s common to bake elements at their highest quality to save on constant rendering. I can't find the exact line since I believe this part of rendering is handled by Lime. But OpenFL does apply some sort of baking of the rendered rect. This can be proven by zooming into a hitbox with the old method and then zooming out. If the hitbox wasn't cached, it should go back to the performance pre-zoom, but it keeps the same low framerate as when zoomed in.

4. At best this alleviates specific cases of bad debug draw performance, there are many that have nothing to do with zoom or off-screen objects

I mentioned zoom because it’s the most common reason for these leaks to happen in the Flixel projects I've worked on. But they're also possible in other common cases, such as trying to render an object that is scaled beyond the bounds of a camera (like an overlay or background sprite).

[Geokureli]

4. At best this alleviates specific cases of bad debug draw performance, there are many that have nothing to do with zoom or off-screen objects

I mentioned zoom because it’s the most common reason for these leaks to happen in the Flixel projects I've worked on. But they're also possible in other common cases, such as trying to render an object that is scaled beyond the bounds of a camera (like an overlay or background sprite).

The point I was trying to make but failed to mention was that I would remove the "Fixes 3164" as this is not a total fix... At least I think it's not but I haven't tested. My guess is that the issue will need to stay open, even after this

[MaybeMaru]

The point I was trying to make but failed to mention was that I would remove the "Fixes 3164" as this is not a total fix... At least I think it's not but I haven't tested. My guess is that the issue will need to stay open, even after this

Ah I see what you mean. Yeah that's why I mentioned this could be considered more of a "temporal" solution while the method Cheems suggested gets worked on further. I'll remove the "fixes" part of the comment.