Skip to content

Coordinator: use render mm_metadata on prefill instead of full kwargs_data #2722

Description

@zhouyou9505

What would you like to be added

After vLLM #54659, render returns features.mm_metadata (metadata-only MultiModalKwargsItem blobs: placeholder_metadata_fields / keep_on_cpu, without encoder tensors such as pixel_values). Prefill may omit kwargs_data and send mm_metadata when ec_transfer_params is set.

Update the coordinator to:

  1. Parse optional mm_metadata from the render response into each MultimodalEntry.
  2. Keep encode on full kwargs_data (ViT still needs pixel tensors).
  3. On prefill, when ec_transfer_params is non-empty and every entry has MMMetadata, emit mm_metadata and omit kwargs_data; otherwise keep sending kwargs_data for older renderers and non-EC paths.
Encode  = kwargs_data
Prefill = mm_metadata + ec_transfer_params
no EC / old renderers = kwargs_data unchanged

Documented as the follow-up to the vLLM half in docs/communication.md (Optimization: avoid sending pixel data to prefill).

Why is this needed

Prefill only needs lightweight multimodal metadata (for example image_grid_thw for mRoPE) plus EC transfer params. Forwarding full kwargs_data re-ships ~1 MB/image of pixel_values that encode already consumed. Dropping those blobs cuts coordinator-to-prefill payload size on E/P/D multimodal traffic.

Success criteria

  • Render stores mm_metadata when present; absent field remains backward compatible.
  • Prefill with NIXL EC + metadata uses mm_metadata and omits kwargs_data.
  • Prefill without EC (or without metadata) still sends kwargs_data.
  • Encode unchanged.
  • Unit tests cover the prefer-metadata and fallback paths.
  • docs/communication.md describes the mm_metadata contract (not the earlier plain image_grid_thw sketch).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions