Is it possible to run it under 8 GB VRAM ?

[Greatz08]

I have NVIDIA RTX 4060 8GB VRAM, so i wanted to know their is anyway to run it in linux environment where no one has shared memory.

[nitinmukesh]

I have the same 4060 Windows laptop and using nunchaku successfully
8 GB VRAM + 16 GB RAM

What do you mean by shared memory?
How much RAM do you have?

[Greatz08]

@nitinmukesh thanks for this valuable information. Btw which video generation model were you able to run under 8 GB VRAM ? As far as i can remember nunchaku currently doesnt support any video generation model yet, so i am interested to understand what and how are you using it :-)) .

wan2.1-flf2v-720p-14b-q2_k.gguf 7.93 GB - this is the lowest size of video generation model i saw on huggingface, its quality might be too bad ? It's size does exceed 8GB VRAM, so if you are using it in windows, then it must be consuming 4 GB additional shared ram in form of vram, because of which you might be able to run it in windows atleast at low t/s, plus you mentioned video upscaler which you would be using to upscale video. This is what i can think of, correct me if i am wrong here.

[nitinmukesh]

@Greatz08

I am regularly using LTX-Video, sometimes 1.3B model of Wan/Vace. Remember it takes time to generate video on such VRAM.

With LTX, I create small videos where only basic animation is required. Difficult to get good results but inference is fast for old model which take only few minutes to generate video.
newgenai79/sd-diffuser-webui#11

And yes I don't even try 14B model as I have only 16 GB RAM so offloading takes very long time.

[Greatz08]

@nitinmukesh Nice, btw have you shared any great generation with these tiny models with anyone ? I am interested to see how much capable these small models are :-))

[Greatz08]

Oopsie i forgot to click the link you provided, so you have showcased the video in that section. Looking nice generation, btw how much time it took for that 6 second generation ?

[nitinmukesh]

Oopsie i forgot to click the link you provided, so you have showcased the video in that section. Looking nice generation, btw how much time it took for that 6 second generation ?

From what I remember it took 4-6 minutes.

[lmxyy]

I have NVIDIA RTX 4060 8GB VRAM, so i wanted to know their is anyway to run it in linux environment where no one has shared memory.

should be runnable with offloading.

[Greatz08]

[M98M]

I have a similar setup and managed to run flux svd int4 with the t5xxl_fp8_e4m3fn_scaled.
The thing that surprised is the massive size of text encoder compared to other modules. So I tried using the AWQ version of T5 but it only works on GPU and got OOM.

Is there a way to use T5 GGUF as text encoder?

[nitinmukesh]

Any reason why you want to use T5 GGUF instead of qencoder?

If you want to use Gradio WebUI, you can try this. I am in process of updating this
https://github.com/newgenai79/sd-diffuser-webui/

If you want to borrow code instead, refer this
https://github.com/newgenai79/sd-diffuser-webui/tree/main/modules/svdquant

[M98M]

Any reason why you want to use T5 GGUF instead of qencoder?

If you want to use Gradio WebUI, you can try this. I am in process of updating this https://github.com/newgenai79/sd-diffuser-webui/

If you want to borrow code instead, refer this https://github.com/newgenai79/sd-diffuser-webui/tree/main/modules/svdquant

apparently for T5xxl, Q8 gguf is better than fp8. even Q6 is not worse. i don't know what qencoder is.
with fp8 95% of my ram is full so it would be nice to save 1-1.5 GB of RAM.
I'm using comfyUI.

[nitinmukesh]

With Comfy you have the option to use qencoder. I don't use Comfy but I have seen someone mentioned the workflow.

Qencoder is quantized text encoder (nunchaku format)
https://huggingface.co/nunchaku-tech/nunchaku-t5