Support soft_prompt or inputs_embeds?

[jessiewiswjc]

Does triton-infernece-server support multi-modal models such as blip2 in trt-llm https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/blip2?

[juney-nvidia]

@jessiewiswjc

Hi, although we haven't provided such example of BLIP2 pipeline in Triton backend repo, the entire inference sequence can be organized in sequential order so there should not be any blockers preventing you from calling the run.py in Triton backend.
Did you meet any specific issue?

June

[jessiewiswjc]

@jessiewiswjc

Hi, although we haven't provided such example of BLIP2 pipeline in Triton backend repo, the entire inference sequence can be organized in sequential order so there should not be any blockers preventing you from calling the run.py in Triton backend. Did you meet any specific issue?

June

I am sorry to hear NVIDIA/TensorRT-LLM#695 (comment). Does the preprocessing in triton not support multi-modal models?