[RyanJDick]

Summary

This PR adds support for the FLUX LoRA model format produced by OneTrainer.

Specifically, this PR adds:

• Support for DoRA patches
• Support for patch models that modify the FLUX T5 encoder
• Probing / loading support for OneTrainer models

Known limitations

• DoRA patches cannot currently be applied to base weights that are quantized with bitsandbytes. The DoRA algorithm requires accessing the original model weight in order to compute the patch diff, and the bitsandbytes quantization layers make this difficult. DoRA patches can be applied to non-quantized and GGUF-quantized layers without issue.
• This PR results in a slight speed regression for a very particular inference combination: quantized base model + LoRA with diffusers keys (i.e. uses the MergedLayerPatch). Now that more LoRA formats are using the MergedLayerPatch, it was becoming too much work to maintain this optimization. Regression from ~1.7 it/s to ~1.4 it/s.

Future Notes

• We may want to consider dropping support for bitsandbytes quantization. It is very difficult to maintain compatibility for across features like partial-loading and LoRA patching.
• At a future time, we should refactor the LoRA parsing logic to be more generalized rather than handling each format independently.
• There are some redundant device casts and dequantizations in autocast_linear_forward_sidecar_patches(...) (and its sub-calls). Optimizing this is left for future work.

Related Issues / Discussions

• This PR should address a handful of the LoRAs reported in [bug]: Certain FLUX LoRAs don't install #7131 (specifically, most of the envy* LoRAs).
• This PR should address the example in [bug]: Model manager can't install Flux LoRa created in OneTrainer #6912 (though the intended effect of that LoRA is not totally clear, so its hard to verify with full confidence).

QA Instructions

OneTrainer test models:

• https://civitai.com/models/844821/envy-flux-dark-watercolor-01?modelVersionId=945159 (DoRA, transformer only)
• https://civitai.com/models/836757/envy-flux-digital-brush-01?modelVersionId=936167 (hada, transformer only)
• ball_flux from [bug]: Model manager can't install Flux LoRa created in OneTrainer #6912 (DoRA, transformer/clip/t5)

The following tests were repeated with each of the OneTrainer test models:

• Test with non-quantized base model
• Test with GGUF-quantized base model
• Test with BnB-quantized base model
• Test with non-quantized base model that is partially-loaded onto the GPU

Other regression test:

• Test some SD1 LoRAs
• Test some SDXL LoRAs
• Test a variety of existing FLUX LoRA formats
• Test a FLUX Control LoRA on all base model quantization formats.

Merge Plan

No special instructions.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Tests added / updated (if applicable)
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)