Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support v-prediction checkpoint models when set in the model manager #7504

Merged
merged 3 commits into from
Jan 15, 2025

Conversation

dunkeroni
Copy link
Contributor

@dunkeroni dunkeroni commented Dec 31, 2024

Summary

Capture prediction type from model settings and pass it to the scheduler to enable v_prediction checkpoints to work again.

Related Issues / Discussions

I'm certain there is a more correct way to handle this in the model manager that would also allow automatic detection from checkpoint files, but this at least gets vpred models working again before we go make sweeping changes to the model probe code.

Since our conversion code does not check the settings on the checkpoint, attempting to convert a v_prediction model to Diffusers will still default over to epsilon prediction and it will forever fail afterwards. It is not currently possible for users to change a diffusers model to v_prediction in Invoke without editing the local json files. User Mewt on discord has found that huggingface repos with correct scheduler_config.json are able to work without issues, but I have found others online that still specify epsilon.

Closes #7495
Closes #6268 (edit by @psychedelicious to get GH to pick up that this PR closes this issue)

QA Instructions

Before:
image

After:
image

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • Documentation added / updated (if applicable)
  • Updated What's New copy (if doing a release after this PR)

@github-actions github-actions bot added python PRs that change python files invocations PRs that change invocations labels Dec 31, 2024
@GrettaKlu
Copy link

Not sure what the right way of doing this is, but the changes here error when using the upscale tab. The file "tiled_multi_diffusion_denoise_latents.py" on line 216 references the get_scheduler() function and needs the unet_config argument passed through as well.

@RyanJDick
Copy link
Collaborator

I added a fix for the Multi-Diffusion inference issue in 160a36a.

I tested the following...

Setup:

  1. Install noobaiXLNAIXL_vPred10Version model
  2. The prediction_type gets probed incorrectly as epsilon (this will have to be addressed in a separate PR)
  3. Change the model's prediction_type in the models tab.

Tests:

  • Generate with the noobai model (v-pred), and verify that it produces coherent results
  • Generate with an epsilon-pred model and verify that the output is the same as before this change.
  • Upscale with the noobai model and verify that it works in a Multi-Diffusion pipeline
  • Upscale with an epsilon-pred model and verify that the output is the same as before the change.

@psychedelicious
Copy link
Collaborator

I believe this also closes #6268.

@hipsterusername hipsterusername enabled auto-merge (rebase) January 15, 2025 13:32
@hipsterusername hipsterusername merged commit 497bc91 into invoke-ai:main Jan 15, 2025
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
invocations PRs that change invocations python PRs that change python files
Projects
None yet
5 participants