LTX-2 condition pipeline

[sayakpaul]

#12915 added support for LTX-2.

What's supported?

• Single stage T2V and I2V
• Upsampling

Similar to the LTX condition pipeline, it would be nice to have support for something similar for LTX-2 as well.

Both the single and double stage pipelines support passing conditions.

[surajyadav-research]

Hi @sayakpaul,
I’m interested in contributing to this feature and would love to work on it. Could you please assign it to me?

[sayakpaul]

Thanks for your interest!

Could you share your plan as to how you want to approach the PR?

[surajyadav-research]

I’m looking into the LTX pipeline and will follow up with a plan ASAP.

[anjaneya-sharma]

@sayakpaul I'm also very much interested in this, and would love to collaborate with @surajyadav-research , if thats alright.

[surajyadav-research]

Hi @anjaneya-sharma
I’m looking into LTX pipeline to come up with a plan for the PR. I’ll put together a plan and share it shortly, and then we can discuss it. Happy to collaborate on this.

[surajyadav-research]

Hi @sayakpaul
I’ve come up with the following plans.

1) New shared module

Contains all conditioning logic so pipelines only call helpers.

In this file

• LTX2VideoCondition
  • if image (I2V) given → add condition at frame_index=0, strength=1.0
• encode_conditions_to_latents(conditions, vae, height, width, num_frames, device, dtype) -> encoded_conditions
  • preprocess/resize → VAE encode → map pixel frame_index → latent_frame_index
• apply_conditions_to_latents(latents, encoded_conditions) -> latents

2) Stage-1 pipelines (denoising): add conditions=None

A) T2V

Modify: pipelines/ltx2/pipeline_ltx2.py
Add conditions=None to LTX2Pipeline.__call__:

• encoded = encode_conditions_to_latents(conds, ...)
• optional pre-loop: latents = apply_conditions_to_latents(latents, encoded)
• inside denoise loop (after scheduler step): latents = apply_conditions_post_step(latents, encoded)

B) I2V

Modify: pipelines/ltx2/pipeline_ltx2_image2video.py
Add conditions=None to LTX2ImageToVideoPipeline.__call__:

• auto frame-0 condition
• same encode + per-step enforcement as T2V

3) Stage-2 Upsampling: distribution-matched conditioning

Modify: pipelines/ltx2/pipeline_ltx2_latent_upsample.py
Add conditions=None to LTX2LatentUpsamplePipeline.__call__:

• encode conditions at input latent scale: encoded_in = encode_conditions_to_latents(...)
• upsample main latents: latents_up = upsampler(latents_in)
• upsample each condition latent with same upsampler - encoded_up
• inject: latents_up = apply_conditions_to_latents(latents_up, encoded_up)
• decode

[sayakpaul]

Thanks for the plan.

Could you elaborate a bit about 1?

Also, I think we can focus only on stage 1 for now.

[surajyadav-research]

Hi @sayakpaul
thanks for reviewing the plan.
This is the single-stage plan I came up with.

1. Create new helper module: pipelines/ltx2/pipeline_ltx2_condition.py

   • Add LTX2VideoCondition dataclass
   • Implement helpers:
     • ensure_conditions_list()
     • encode_conditions_to_latents() -> returns (cond_latents_packed, cond_mask_packed)
     • apply_conditions_to_latents() -> blends/injects into latents
     • expand_mask_for_cfg() -> duplicates mask for CFG batch
     • prepare_noise_latents_packed() (used by I2V to start from noise)

2. Modify LTX2Pipeline.__call__ (T2V)

   • Add arg: conditions=None
   • After prepare_latents(): encode conditions + inject once
   • In denoise loop:
     • Use per-token timestep: video_timestep = t * (1 - cond_mask_cfg)
     • After scheduler step: re-inject latents = apply_conditions_to_latents(...)

3. Modify LTX2ImageToVideoPipeline.__call__ (I2V)

   • Add arg: conditions=None
   • Replace special “frame0-only” I2V logic:
     • Start from noise latents (prepare_noise_latents_packed)
     • Add implicit condition: input image as frame_index=0 (unless user provided it)
     • Encode+inject conditions same as T2V
   • In denoise loop:
     • Same masked timestep + re-inject after scheduler step

[sayakpaul]

Plan looks good but let's keep LTX2ImageToVideoPipeline untouched for now and instead, do LTX2ConditionPipeline. We could also follow the LTXConditionPipeline and pack the conditions into a dataclass like so:

diffusers/src/diffusers/pipelines/ltx/pipeline_ltx_condition.py

Line 96 in 3981c95

class LTXVideoCondition:

That is a good design and it would be nice to stick to it.

[surajyadav-research]

Hi @sayakpaul
Thanks for reviewing the plan. I’ll come up with an updated plan.

[surajyadav-research]

Hi @sayakpaul
thanks for reviewing the plan, I updated plane now its similar to LTXConditionPipeline

1) New file

Create: diffusers/pipelines/ltx2/pipeline_ltx2_condition.py

Add:

• @dataclass LTX2VideoCondition(image=None, video=None, frame_index=0, strength=1.0)
• class LTX2ConditionPipeline(...same modules as LTX2Pipeline...)
• def __call__

2) Parse + preprocess (inside __call__)

Add helpers:

• _normalize_condition_inputs
• _trim_conditioning_sequence
• _preprocess_conditioning_tensors

3) Encode conditions into LTX-2 video latents

Add helpers (copied/adapted from LTX2Pipeline):

• _retrieve_latents(...)
• _normalize_latents(...)
• _pack_latents(...) / _unpack_latents(...)
• _denormalize_latents(...)

Core:

• prepare_latents_with_conditions

Rules:

• frame_index == 0: encode→normalize, blend into base latent frames via torch.lerp(...), set mask for those tokens
• frame_index > 0: encode→normalize→pack as extra tokens, prepend to latents, mask = strength, track extra_tokens

4) Timesteps + denoise_strength (LTX-2 schedule)

Reuse LTX-2:

• calculate_shift(...), retrieve_timesteps(...), audio_scheduler = copy.deepcopy(self.scheduler)
  Add:
• get_timesteps(..., denoise_strength)

If denoise_strength < 1: compute latent_sigma and mix provided latents with noise like LTX-1 (edit mode).

5) Denoising loop (freeze conditioned tokens)

Add:

• add_noise_to_image_conditioning_latents(...)

Loop:

• run transformer exactly like LTX2Pipeline (video+audio, CFG)
• scheduler step video/audio
• freeze tokens:
  • tokens_to_denoise = (t_norm < (1.0 - conditioning_mask)).unsqueeze(-1)
  • latents = torch.where(tokens_to_denoise, denoised_latents, latents)

6) Cleanup + decode

• latents = latents[:, extra_tokens:]
• _unpack_latents → _denormalize_latents → vae.decode
• audio path unchanged (same audio_vae.decode + vocoder as LTX2Pipeline)

7) Other files to change

• diffusers/pipelines/ltx2/__init__.py: export LTX2ConditionPipeline, LTX2VideoCondition

[sayakpaul]

Okay plan looks reasonable to me. Thanks for iterating!