Complex Inpaint Masks

[dunkeroni]

Summary

This PR adds two new core features to inpaint masks on the canvas.

1. Inpaint masks can have independent Denoise Limit values that reduce the strength of the mask. When multiple masks overlap, the strongest is used. It leverages the same tech as our compositing, so all denoise limits are capped at the global Denoising Strength of the generation.
2. Inpaint masks can now insert image noise to the input before image-to-latents is called. This greatly improves variation and textures on inpaint operations, especially with drawn inputs. This was previously available in a filter, but was cumbersome to apply to inpaint operations.

Now there is a reason to use more than one mask layer :)

Some back-end changes that were required to make it work:

• Create Gradient Mask node now accepts grayscale inputs for expansion (previously assumed boolean input). This uses a custom 2D filter built on Torch. To keep processing times reasonable, the gradient mask is calculated at the latent scale (1/8) instead of the image scale now. Since it gets resized to latent scale during denoising anyway, this should not cause any impact on quality.
• Image Noise node now accepts an optional mask to limit where noise is applied, and greyscale values on this mask can limit the strength of the noise (which is how the canvas graph uses it).
• addInpaint/addOutpaint canvas graph builders no longer use AlphaToMask node for the inpaint masks. There is a new compositing function in the frontend that combines inpaint masks directly into a greyscale mask for denoise limits and image noise. Reduces graphs by 1 intermediate image for normal inpaint operations.

Mask Noise Demonstration:
https://github.com/user-attachments/assets/c690ab72-b7ba-435e-b556-cd91ddf2d527

Inpaint Mask Noise improves Flux as well:

Multiple Denoise Limits Demonstration:
https://github.com/user-attachments/assets/d3e33229-2e83-4157-bb31-300057000597

Related Issues / Discussions

Denoise Limits do not work on flux because flux is not yet compatible with gradient masks. Currently it thresholds at 100% and 50% instead. It doesn't break anything, but using a limit below 100% has almost no effect on the image. Works now, but isn't super helpful because of how flux works.

I am a neophyte at frontend code, and I had a lot of conversations with the Q Chat AI to add these features. So if anything in the code looks dumb, then that'd be because I'm dumb.

QA Instructions

I have tested inpaint/outpaint/both on SDXL/Flux at 1024/1536 resolutions with/without scaling, with/without mask noise, and with/without denoise limits. That should cover all the bases where this could break things, but be on the lookout for anyone reporting some new and slightly out of ordinary results on canvas.

Merge Plan

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Tests added / updated (if applicable)
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

[hipsterusername]

Good addition (especially the noise!)

I'm a bit fearful of complicating the Inpaint Mask UX in two ways:

• Making it a larger/more cluttered item in the stack of layers (Potential Solve: Establishing some other way to enable the feature vs large button)
• Making it inconsistent between different model architectures (Although, this is less relevant if the first is solved)

Let us do some noodling on the UX here

[dunkeroni]

I did have the same worry, which is why they are both also in the right click menu. Currently it follows the same pattern and layout as regional guidance layers, where all buttons go away when one is clicked, and additional controls are added by right click.

Flux can be pretty easily changed to handle gradient masks, since it already does a 2-stage version. Cogview/SD3 can be similarly updated. Low-step models like Schnell might behave oddly (LCM did originally), so I'll have to do a bit of testing and that'll be a separate PR.

We can remove the buttons and put the upfront inpaint mask size back to where it was. That condenses the UX, but would almost entirely drop discoverability for what are (in my opinion) frequently helpful and easily approachable controls. Things hidden in menus almost become a secret technique rather than a queue for what the user should try. We could start with that, and if either slider becomes a thing that users interact with a lot and want better exposed then we add the buttons back in. Alternative would be to make the button visibility a 13th checkbox in the canvas settings menu, but that feels clunky and out of place.

[dunkeroni]

Update: The other architectures all share the same _apply_mask_gradient_adjustment() code, so it was easier than I thought to make them compatible with mask denoise limits. As a bonus, flux et al actually behave correctly with our compositing methods now, so it produces slightly cleaner edges when inpainting normally.

At least for flux dev, attempting to use denoise limits on masks is hit or miss since so much of the final image is decided in the first few steps. Adding noise helps that a bit, but the whole usable range is still somewhere between 0.7-0.9