add Bernini-R notebook#3482
Open
openvino-dev-samples wants to merge 20 commits into
Open
Conversation
ByteDance/Bernini-R-1.3B-Diffusers is a unified multi-task diffusion renderer
(t2i / i2i / t2v / v2v / r2v / rv2v) fine-tuned from Wan2.1-1.3B. Its generation
logic (per-step loop, 7 guidance modes, source-id rotary embeddings, per-step
token assembly) is data-dependent python that a static OpenVINO graph cannot
express, so only the leaf compute is exported to OpenVINO while the original
bernini sampler/pipeline code drives it unchanged:
- text encoder (UMT5) -> single static graph
- transformer -> BlocksCore (condition embedder + blocks + proj) with
patch-embedding and rotary construction kept in torch;
packed-token axis is dynamic so one graph serves every
guidance combo. varlen attention -> SDPA, complex RoPE
-> real cos/sin to stay OpenVINO-friendly.
- VAE encoder/decoder -> one graph per temporal latent length (the Wan VAE
walks frames with a causal feature-cache loop), built
eagerly for images and lazily for video lengths.
Verified against the reference: transformer split is bit-exact, VAE ~1e-6;
full t2i/t2v/r2v run end to end; INT8 matches FP16. On GPU/NPU the UMT5 text
encoder and VAE run in fp32 (fp16-overflow -> black image for some prompts)
while the heavy transformer stays in fast fp16, keeping GPU output aligned with
CPU at ~2x the speed of all-fp32.
Files: ov_bernini_helper.py (conversion + OpenVINO wrappers + load_ov_pipeline),
gradio_helper.py (task-routed demo), bernini-r-image-video.ipynb, README.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Align the task -> guidance_mode mapping with the reference bytedance/Bernini testcases (t2i/t2v use t2v_apg, i2i uses v2v, v2v uses v2v_apg) and add the per-task system prompts and required-input table (TASK_SYSTEM_PROMPT / TASK_INPUTS). The system prompt is prefixed to the user prompt as in the reference pipeline. The original model supports image editing (i2i) and video editing (v2v); both now run end to end on the real checkpoint (i2i std ~64; v2v produces a valid mp4). The notebook gains dedicated i2i and v2v cells (downloading the reference source image/video) and the gradio demo routes inputs per TASK_INPUTS. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…E device) - Default video to 20 frames and 4 steps in the notebook and gradio demo, as in the wan2.1 / wan2.2 notebooks. - Add num_frames_to_latent() helper (Bernini rounds num_frames to n//4*4+1, the VAE then uses (n-1)//4+1 latent frames; 20 -> 6) and pre-convert that VAE graph so the default video length needs no on-demand conversion. Other lengths are still converted and cached on first use. - Place the VAE on CPU by default: decoding a multi-frame video allocates a large USM buffer that can exhaust GPU memory (observed "Can not allocate ... USM Device"); the VAE is one forward per generation, so CPU costs little while the transformer / text encoder keep running on GPU. Documented in the README and the device-selection cell. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ter startup Addresses three issues: 1. load_ov_pipeline no longer takes the original model path. convert_pipeline now copies the bernini/vae configs (plus tokenizer/scheduler) into the OpenVINO output dir, so the converted model is self-contained. 2. load_ov_pipeline reads everything from the OV dir and loads NO PyTorch model weights -- only OpenVINO IR is compiled. The original dir is needed only via the optional source_model_dir, and only to convert a VAE graph for a video length that was not pre-built. (The lone torch tensor kept at run time is the few-KB patch-embedding Conv3d that drives the data-dependent patchify/rotary.) 3. The long pause before the denoise progress bar was the GPU compiling the ~5 GB fp32 UMT5 text encoder (~80 s) and thrashing GPU memory. The text encoder and VAE now default to CPU (TE must be fp32 anyway; it compiles in ~2 s on CPU and runs twice per generation), while the transformer keeps using the GPU. If TE/VAE are placed on GPU, an fp32 hint is applied automatically. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The VAE decode is ~6.7x faster on GPU for a single image frame (~1 GB) but a multi-frame video decode unrolls the Wan causal-conv loop into a tens-of-GB USM buffer (~24 GB at 13 frames, ~34 GB at 21) that exhausts the integrated GPU once the transformer is also resident -- confirmed on both stable 2026.2 and nightly 2026.3 OpenVINO (the isolated decode can spill to host, but the full pipeline still OOMs). OVVAEWrapper now chooses the device per call by latent temporal length: lengths <= vae_gpu_max_latent_frames (default 1, i.e. images) run on the requested GPU device, longer video lengths transparently use a CPU-compiled copy of the graph. The per-length graph cache is keyed by (length, device) and the chosen device is logged once. The notebook's VAE device widget now defaults to AUTO since the wrapper self-selects. Verified: image t2i with VAE on GPU (std 46.77), 20-frame t2v auto-routes the VAE to CPU and produces a valid mp4 with no OOM. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per request, the VAE simply runs on whatever device it is assigned (default AUTO) -- the size-based GPU->CPU routing for video is removed. Correctness is preserved by the existing fp32 precision hint applied on GPU/NPU; verified the VAE on GPU is not black (matches CPU, maxdiff ~2e-3) for images. Memory/OOM for large video decodes is out of scope. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
Debugged the VAE-on-GPU black-image concern: with a real pipeline latent the VAE decodes correctly on GPU in fp16 for images on stable OpenVINO, and for video too on the 2026.3 nightly (its GPU plugin spills the large multi-frame activation buffer to host instead of failing). fp16 vs fp32 max diff is ~0.013, no NaN, not black. So the VAE now runs in fp16 on GPU (was fp32) and the notebook installs the OpenVINO nightly. The UMT5 text encoder remains fp32 -- it is the genuine fp16 overflow source and stays on CPU by default. Verified end to end with the shipped default config on nightly: t2i (VAE GPU fp16) std 34.06 not black; 20-frame t2v (VAE GPU fp16) produces a valid mp4. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes the CI spellcheck failure for notebooks/bernini-r-image-video: ByteDance, renderer, VTI, uncond, patchify, UMT, bernini, rv, testcases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The truststore line was a workaround for a TLS-intercepting corporate proxy on the dev machine, not something a general user needs. Remove it from the shipped notebook. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
openvino-dev-samples
force-pushed
the
bernini-r-1.3b
branch
from
7636770 to
fd6a558
Compare
openvino-dev-samples
force-pushed
the
bernini-r-1.3b
branch
from
5173e69 to
642f712
Compare
Validated the notebook from a fresh venv on a tiny model and hit two install breakages: - the upstream Bernini HEAD now hard-imports `veomni` (heavy multi-GPU dep) and fails to import for OpenVINO inference -> pin commit 94a52f78. - modelscope pulls huggingface-hub>=1.0, which breaks transformers 4.57.3 (ImportError) -> pin huggingface-hub<1.0 after it. Only the install cell changed; convert + t2i + t2v verified end to end in the clean env on the tiny model. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
pip skips reinstalling git packages when the version (0.1.0) already matches, so an environment that previously installed Bernini HEAD keeps the veomni-dependent build and still fails with ModuleNotFoundError: veomni. Add --force-reinstall so the pinned commit replaces it. Verified HEAD -> pinned switch in a clean venv. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The device-selection cell uses device_widget but the prerequisites cell only imported collect_telemetry, raising NameError. Add device_widget to the import. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
|
This PR will be closed in a week because of 2 weeks of no activity. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.