Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make T5 models interchangeable between FLUX/SD3 workflows #7544

Merged
merged 1 commit into from
Jan 15, 2025

Conversation

RyanJDick
Copy link
Collaborator

@RyanJDick RyanJDick commented Jan 9, 2025

Summary

Prior to this change, attempting to use a T5 model that was bundled with an SD3 base model in a FLUX workflow resulted in the following exception.

[2025-01-09 22:58:53,711]::[InvokeAI]::ERROR --> Error while invoking session 3f4f9287-bb23-44f7-8e52-bbf6c4b114e8, invocation c3bd3f6d-71cd-4b86-a4c2-8e43758e4185 (flux_text_encoder): 
[2025-01-09 22:58:53,711]::[InvokeAI]::ERROR --> Traceback (most recent call last):
  File "/home/ryan/src/InvokeAI/invokeai/app/services/session_processor/session_processor_default.py", line 129, in run_node
    output = invocation.invoke_internal(context=context, services=self._services)
  File "/home/ryan/src/InvokeAI/invokeai/app/invocations/baseinvocation.py", line 300, in invoke_internal
    output = self.invoke(context)
  File "/home/ryan/.pyenv/versions/3.10.14/envs/InvokeAI_3.10.14/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/home/ryan/src/InvokeAI/invokeai/app/invocations/flux_text_encoder.py", line 60, in invoke
    t5_embeddings = self._t5_encode(context)
  File "/home/ryan/src/InvokeAI/invokeai/app/invocations/flux_text_encoder.py", line 78, in _t5_encode
    assert isinstance(t5_text_encoder, T5EncoderModel)
AssertionError

After this change, all T5 models can be used interchangeably for both FLUX and SD3 workflows.

Related Issues / Discussions

QA Instructions

I tested the following combinations:

  • FLUX text-to-image, standalone T5
  • FLUX text-to-image, SD3-bundled T5
  • SD3 text-to-image, SD3-bundled T5
  • SD3 text-to-image, standalone T5
  • SD3 text-to-image, BnB-quantized T5

Merge Plan

No special instructions.

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • Documentation added / updated (if applicable)
  • Updated What's New copy (if doing a release after this PR)

@github-actions github-actions bot added python PRs that change python files invocations PRs that change invocations backend PRs that change backend files labels Jan 9, 2025
@hipsterusername hipsterusername force-pushed the ryan/make-t5-interchangeable branch from 7bc7aae to a218957 Compare January 15, 2025 21:18
@psychedelicious psychedelicious merged commit b301785 into main Jan 15, 2025
15 checks passed
@psychedelicious psychedelicious deleted the ryan/make-t5-interchangeable branch January 15, 2025 21:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend PRs that change backend files invocations PRs that change invocations python PRs that change python files
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[bug]: Assertion error related to text_encoder while trying to generate with Flux Dev Quantized
3 participants