SeedVR2 color shift

[urixl]

I've seen this bug before in the discussions and now I've stumbled upon it by myself.
SeedVR2 likes to create oversaturated colors in some cases, like faces or color accents.
Let's see the comparison:
Original:

Seed2VR-> Topaz Starlight Mini as a touch-up.

Topaz Starlight Mini

Look at the lips of the woman, a flare next to her head, a cocktail on the desk. Look at the color of her cheeks.
That's a lot of hallucinations.

Next shot:
Original:

Seed2VR-> Topaz Starlight Mini as a touch-up.

Topaz Starlight Mini

Her lip gloss was not that red, her skirt was more dull red.

More nuanced shot.

Original:

Seed2VR-> Topaz Starlight Mini as a touch-up.

Topaz Starlight Mini

Again, Topaz Starlight Mini yields accurate colors while SeedVR2 tries to enrich colors and push contrast to match "AI-like" beautiful picture.

[adrientoupet]

Thank you for sharing some pictures - it's a known issue of SeedVR2. You have different color correction mode in the UI to play with. You can also try a post upscale grade to match the input more closely (eg Kijai's color match)

I'll see if there is anything I can do in the implementation to improve things, but it's not high on my priority list at the moment. Thanks

[mindplay-dk]

Likely happened while they were training the model, right? A flaw in the process when source material was scaled down? Any variation in colors or contrast etc. between the input and the resized video (which decoders might cause when converting between color models etc.) and the model would have learned to do this, right?

[naxci1]

Actually, the color contrast is too much here, I don't think this is a big problem, the real problem is that SLM can't preserve the details, this is what I always say, SLM and SLS have never been able to preserve the details, they distort it, they distort the face and the details on the face and they make it up themselves with artificial intelligence, they make the texts completely alien text, now you can see for yourself the difference in quality of both images.

[urixl]

the color contrast is too much here, I don't think this is a big problem

This is a big problem.
I don't need my footage to be beautified.
You see - there is no constant color shift or over-saturation which can be mathematically corrected.
It's a beauty filter and false color accents I don't need.

You have different color correction mode in the UI to play with. You can also try a post upscale grade to match the input more closely (eg Kijai's color match)

It's not a color grading problem, there's no LUT I can apply post-conversion. Most colors are represented correctly. I can't turn down reds for example, it'll affect skin color too.
How would you remove red lips for example?

[naxci1]

SeedVR2 is for video restoration; you'll need to perform other operations in other video editors. I perform the color correction in Sony Vegas; the important thing is the video restoration. Each application has its own functions, and no single application can do it all.

[urixl]

It does not restore video if it beautifies it.
Another examples:

Look at the reds saturation. Why people have so bright red ears for example?

What are the yellow blobs that weren't so visible before? This video has to be manually retouched to get rid of these blobs.

[naxci1]

What are your settings? These problems occur when the batch size is low. I used to have color problems when I upscaled 1.5x. Now, when I upscaled 3x, the color problems were completely resolved. These are the settings I use.

[naxci1]

Parti boyutu 117'dir. Yeni boyut 640 yükseklik, vae karo boyutu 320, takas edilecek blok sayısı 20'dir.

Send me a picture of your workflow

[urixl]

Parti boyutu 117'dir. Yeni boyut 640 yükseklik, vae karo boyutu 320, takas edilecek blok sayısı 20'dir.

Send me a picture of your workflow

[urixl]

Here is the original video:
https://drive.google.com/file/d/18B0_BbzHZEq_RAvm0qZZDuBsPDTh5ABV/view?usp=drive_link

[naxci1]

Here is the original video: https://drive.google.com/file/d/18B0_BbzHZEq_RAvm0qZZDuBsPDTh5ABV/view?usp=drive_link

I couldn't download the video. You need to give everyone permission.

There are actually problems with your workflow. Let me download your video, I'll try it.

[urixl]

I couldn't download the video. You need to give everyone permission.

Fixed it, thanks.

There are actually problems with your workflow. Let me download your video, I'll try it.

I know there's a problems but I know nothing about gguf for example - what is it and what are the differences between different models.

[naxci1]

act_1_ses_00004.mp4

act_1_ses_00002.mp4

act_1_ses_00003.mp4

--

On the left is the original video, the others were done with different color corrections. I upscaled it 3x. There's already color distortion in the original video, and it's visible in the pixels when you enlarge it, so it becomes even more noticeable when you enlarge it. My advice to you is to edit the colors with any video editor app after upscaling your video 3x.

[urixl]

I see no difference between color corrections unfortunately.
Can you share which corrections did you use?

[naxci1]

Renk düzeltmeleri arasında bir fark göremiyorum maalesef. Hangi düzeltmeleri kullandığınızı paylaşabilir misiniz?

I tried 3 different color schemes of SeedVR2.

[naxci1]

My workflow was like this: I first reduced the video by 50%, then upscaled it 3x, and set the batch_size to 45 to avoid OOM errors. I only set it to 180 frames so it would finish faster. You can make the entire video with these settings if you want.

[urixl]

I see you're using 3b-Q8_0.gguf model.
I downloaded it to \ComfyUI_windows_portable\ComfyUI\models\SEEDVR2
But I get the error "Cannot load seedvr2_ema_3b-Q8_0.gguf
GGUF library is required but not installed.
Please install it with: pip install gguf"

I did "pip install gguf", restarted ComfyUI and get the same error.

[naxci1]

3b-Q8_0.gguf modelini kullandığınızı görüyorum. \ComfyUI_windows_portable\ComfyUI\models\SEEDVR2 dizinine indirdim ancak "seedvr2_ema_3b-Q8_0.gguf yüklenemiyor . GGUF kütüphanesi gerekli ancak yüklü değil. Lütfen şunu kullanarak yükleyin: pip install gguf" hatası alıyorum.

"pip install gguf" komutunu kullandım, ComfyUI'yi yeniden başlattım ve aynı hatayı aldım.

\ComfyUI\python_embeded

You need to go into this directory, open a terminal, and run that command.

After doing this, close comfyui completely and open it again.

[urixl]

Unfortunately I get an error
"Fatal error in launcher: Unable to create process using '"D:\a\ComfyUI\python_embeded\python.exe" "D:\Programs\ComfyUI_windows_portable\python_embeded\Scripts\pip.exe" install gguf': ?? ??????? ????? ????????? ????.
PATH variable contains both "D:\Programs\ComfyUI_windows_portable\python_embeded" and "D:\Programs\ComfyUI_windows_portable\python_embeded\Scripts" paths. Where "D:\a\ComfyUI\python_embeded\python.exe" comes from - I don't know.

[naxci1]

pip install gguf

Open a terminal in the D:\Programs\ComfyUI_windows_portable directory and enter this command.

.\python_embeded\python.exe -m pip install gguf

[urixl]

Thank you!
It worked finally.
I found out that all instances of "d:\a\comfyui" in embedded python files, 3898 .pyc and .pyd files.
ComfyUI was downloaded from the official site and obviously authors placed it to thier "d:\a" folder.

[naxci1]

[naxci1]

Output codec I usually use H265, AV1 is also good actually.

[urixl]

After applying BSKL Ai Color Match plugin for Adobe Premiere I can get more accurate results:

Original

Straight from Seedvr2:

After AI color match correction:

It doesn't negate red accents, but corrects overall color and contrast to the original.

Another example, look at the red ear, nose, soles and line:

Original:

Straight from Seedvr2:

After AI color match correction:

The good thing is that I can save the results of plugin as a LUT and apply it fast.

[adrientoupet]

Hi @urixl , @mindplay-dk, @naxci1

Thank you for the thread and example pictures. I've implemented a few new color fix techniques in SeedVR2. I've made LAB the new default as it gives the best results and avoid over-saturation. It's not perfect as I'm trying to compensate what the model does by default, but it's way better than before. See below

Original input:

Wavelet color fix (current seedvr2 implementation):

LAB color fix (new default in the next seedvr2 release):

I'll leave it there for now, it will be deployed in the next big release with all the other stuff. Let's see what you all do with it and then we can improve it further if needed. Thanks

[urixl]

Thank you, LAB looks better that wavelet.