Frame ghosting

[AIisCool]

As I'm sure we've all seen, despite AMAZING results, SEEDVR2 does have an issue; whether it's hardcoded in the model or something to correct via code: when there is medium to fast motion between frames, you can see ghosting/afterimage of the previous frame which leads to distortions for really fast motion. I can also see a similar ghosting/distortion issue with Topaz's Starlight model.

So the point of this discussion is: What is causing it and how do we correct it if such a task is possible?

[naxci1]

The batch_size should be as large as possible. For example, it used to be 101 for me, but I've now set it to 121, and these problems don't occur. I don't recommend going below 101. Depending on your system hardware, the higher you set it, the better the results you'll get. I also set Taled_vae to 768. This is also important; a higher number results in better quality and faster processing.

[naxci1]

[AIisCool]

@naxci1 Tried those exact setting and the issue is still there for me.

Input:

Output:

You can see his left arm is distorting with the plant behind him. If you wish to try with that video, I uploaded it here.

[naxci1]

The link did not work. If the video is corrupted like this, SeedVR2 cannot fix it unfortunately. However, if the video is not corrupted like this and SeedVR2 corrupts it, I think it can be fixed. Upload the original video again.

[AIisCool]

Not sure why the link did not work but I tried again.

No corruption in the input video, it only becomes distorted after processing.

[naxci1]

input video:

The hand is already moving out of place in the intro video. Is this a low-fps video?

On mega.nz, it only shows uploading; I guess there's no download permission.

[naxci1]

It would have been more useful to deinterlaced the video beforehand.

[AIisCool]

No, the FPS is 23.976. As for deinterlacing it's from a Blu-ray download that I resized the resolution down by 50%. That said I have seen the same distortion issue with pure progressive video so interlacing isn't the cause.

Not sure why mega isn't working but I tried proton this time.

[naxci1]

I downloaded the video, deinterlaced it, but it didn't work. The video was corrupted when re-rendered. It would have been easier if it were the original video. I'll run tests with it and let you know the results, but if it were the original video, the DVD version, this problem wouldn't have happened.

[naxci1]

What is perceived as distortion as the hand moves in the image is actually a common video and photography phenomenon known as "motion blur."

[naxci1]

https://yandex.kz/video/preview/3174338796135403253

I also found the 1080p version of the movie, the same scene has the same problem, since the DVD ripping is done in lower quality, all videos have this problem and it cannot be fixed.

[AIisCool]

I don't think that's the case.

Yes there is motion blur, but in the SEEDVR2 video there is green distortions that wasn't in the original:

I tried another video. You can see the pattern on her skirt is all distorted after SEEDVR processing. Yes there is motion and it's blurry as she moves, but in the original the pattern isn't distorted.

Input:

SEEDVR2:

[mindplay-dk]

I don't think that's the case.

Yes there is motion blur, but in the SEEDVR2 video there is green distortions that wasn't in the original:

Huh, now in this example I definitely see what you're talking about - it's as though it's painted the leaves from the plant onto his arm.

If I had to guess, I'd say the input is probably too blurry for the model to make sense of, and so it makes a "wrong best guess".

On the skirt in the other example, I don't see a problem - just looks like motion blur in the input and the output, right?

[naxci1]

share workflow

[naxci1]

That's all I could fix. Unfortunately, it can't fix motion blur. Topaz couldn't fix it either, so I tried that too. It would be nice if this problem was added to the new model.

6_apo8_nyx3_yeni_00002.mp4

[naxci1]

Actually, the title needs to be changed, the problem is "motion blur"

[AIisCool]

Here is the clip I used with the skirt: https://drive.proton.me/urls/9KR6XDHWK8#IGaFvCez57Lg

Have to go for a while, be back later.

[AIisCool]

@adrientoupet curious to know your thoughts on this. Am I misinterpreting motion blur or are distortions happening?

[adrientoupet]

Apologies @AIisCool I'm not sure - and I won't look into this right now. I'm currently focused on finishing the release I'm working on, to clear a lot of of other tickets that have been opened for a while. Once that's done, I'll go back to that thread/your question and will help troubleshoot, if you all don't find a solution before me. Thanks for your patience.

[AIisCool]

No worries! I am always impressed by how much you've improved SeedVR2 in such a short time.

[adrientoupet]

It would be even more impressive if it had been officially released and I had time to make an updated video about it to document everything :) but thank you, appreciate the support... hopefully we'll get there soon!

[AIisCool]

You're welcome, I wish I was far more knowledgeable and able to assist.

[AIisCool]

Hi @adrientoupet I had a thought about the distortions. When I tried upscaling a video using UltimateSDUpscale and tied in the WAN 2.2 model and it's VAE, it didn't have any distortions. I wonder if this may help, if it's possible to somehow integrate WAN's temporal consistency. This is of course assuming the distortions can't be fixed via coding and isn't just part of the SEEDVR model.

[naxci1]

@AIisCool

Could you share your workflow? It might be due to the WAN model difference.

[adrientoupet]

Thank you @AIisCool - I suspect the distortions should be fixable in the code, but I'll report back once I had a chance to dive deeper into it.
Out of curiosity, how does ultimateSDUpscale compare to SeedVR2 ?