What would you like to be added
After vLLM #54659, render returns features.mm_metadata (metadata-only MultiModalKwargsItem blobs: placeholder_metadata_fields / keep_on_cpu, without encoder tensors such as pixel_values). Prefill may omit kwargs_data and send mm_metadata when ec_transfer_params is set.
Update the coordinator to:
- Parse optional
mm_metadata from the render response into each MultimodalEntry.
- Keep encode on full
kwargs_data (ViT still needs pixel tensors).
- On prefill, when
ec_transfer_params is non-empty and every entry has MMMetadata, emit mm_metadata and omit kwargs_data; otherwise keep sending kwargs_data for older renderers and non-EC paths.
Encode = kwargs_data
Prefill = mm_metadata + ec_transfer_params
no EC / old renderers = kwargs_data unchanged
Documented as the follow-up to the vLLM half in docs/communication.md (Optimization: avoid sending pixel data to prefill).
Why is this needed
Prefill only needs lightweight multimodal metadata (for example image_grid_thw for mRoPE) plus EC transfer params. Forwarding full kwargs_data re-ships ~1 MB/image of pixel_values that encode already consumed. Dropping those blobs cuts coordinator-to-prefill payload size on E/P/D multimodal traffic.
Success criteria
What would you like to be added
After vLLM #54659, render returns
features.mm_metadata(metadata-onlyMultiModalKwargsItemblobs:placeholder_metadata_fields/keep_on_cpu, without encoder tensors such aspixel_values). Prefill may omitkwargs_dataand sendmm_metadatawhenec_transfer_paramsis set.Update the coordinator to:
mm_metadatafrom the render response into eachMultimodalEntry.kwargs_data(ViT still needs pixel tensors).ec_transfer_paramsis non-empty and every entry hasMMMetadata, emitmm_metadataand omitkwargs_data; otherwise keep sendingkwargs_datafor older renderers and non-EC paths.Documented as the follow-up to the vLLM half in
docs/communication.md(Optimization: avoid sending pixel data to prefill).Why is this needed
Prefill only needs lightweight multimodal metadata (for example
image_grid_thwfor mRoPE) plus EC transfer params. Forwarding fullkwargs_datare-ships ~1 MB/image ofpixel_valuesthat encode already consumed. Dropping those blobs cuts coordinator-to-prefill payload size on E/P/D multimodal traffic.Success criteria
mm_metadatawhen present; absent field remains backward compatible.mm_metadataand omitskwargs_data.kwargs_data.docs/communication.mddescribes themm_metadatacontract (not the earlier plainimage_grid_thwsketch).