How to integrate Multi-LoRA Setup at Inference with NVIDIA Triton / TensorRT-LLM? I built the engine...

[JoJoLev]

I built the engine, and had two separate LoRA layers with the base llama3.1 model. The output from the build is rank0.engine, config.json, and then a lora folder with the following structure:
lora
|
|>0
| |_> adapter_config.json
| |> adapter_model.safetensors
|
|>1
| |> adapter_config.json
|__ |> adapter_model.safetensors

Is this expected? I figured there would be rank engines? I passed these in the lora directory on the engine build:
trtllm-build --checkpoint_dir ./tllm_checkpoint_1gpu_tp1 --output_dir /opt/tensorrt_llm_engine --gemm_plugin auto --lora_plugin auto --max_batch_size 8 --max_input_len 512 --max_seq_len 562 --lora_dir "/opt/lora_1" "/opt/lora_2" --max_lora_rank 8 --lora_target_modules attn_q attn_k attn_v

Any advice is appreciated.

[syuoni]

Hi @JoJoLev ,

I suppose the output folder is expected. You built the eigine with TP=1, and there was one rank0.engine. The LoRA weights are saved in adapter_model.safetensors under each LoRA folder.

[JoJoLev]

Hi @syuoni

Thanks for the response. Yes, I have a rank0.engine file, and config. My question now is that when I deploy on to a container, say NVIDIA Triton, do I have to include the LoRA weights? Or have those been baked in to the rank0.engine?

[syuoni]

Hi @syuoni

Thanks for the response. Yes, I have a rank0.engine file, and config. My question now is that when I deploy on to a container, say NVIDIA Triton, do I have to include the LoRA weights? Or have those been baked in to the rank0.engine?

Yes, you have to include the LoRA weights. They are not baked into the engine because TRT-LLM supports multi-lora, so has to load LoRA weights dynamically at runtime.

[JoJoLev]

@syuoni got it!

Thank you! So, after running my engine build I had the aforementioned folder structure, if I was to deploy on NVIDIA Triton, would I include the LoRA weights in the 1/ sub folder where my rank0.engine file and config.json are? Or would I be placed on a different path?
I believe this is the container we are going with on deployment.

[syuoni]

Hi @JoJoLev ,

The lora weights under the engine folder (i.e., lora/0/ and lora/1/) are used for ModelRunner(Cpp). For example, run.py script.

To use lora for Triton server, we need to convert the lora weights to the format required by C++ runtime. Please follow the steps shown in this doc. Also, I'd like to remind that the Triton server deployment need to be in the container provided by TensorRT-LLM Backend, see here.

[JoJoLev]

@syuoni , got it. I just converted the LoRA files to the format with tensorrt-LLM. I now have the engine build with lots and have converted the Lora.

When constructing the triton, where do the files go?

Typically with triton there is a structure of:

Path
|
|>1->rank0.engine
| |_>config.json
|
|> config.pbtxt

Where would the lora files go?

[syuoni]

Please see this doc: https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/docs/lora.md#launch-tritonserver

Once lora checkpoints are converted by hf_lora_convert.py, they can be passed to inflight_batcher_llm_client.py via --lora-path

[JoJoLev]

Hi @syuoni

Thanks for the support!
For an update, this works, got the inference running per the tutorial you shared.

Now, the thing I need to configure is how to properly construct this setup to deploy on Sagemaker. For the config.pbtxt in NVIDIA Triton, would I call the Lora paths there? Or would they be in file structure for triton when I deploy endpoint?

Thanks!

[syuoni]

Hi @JoJoLev ,

I don't think I've fully understood your questions. I would say that lora paths are not closely related to config.pbtxt.

Once lora checkpoints are processed by hf_lora_convert.py, the processed lora paths can be used for Triton. They should be specified via --lora-path to inflight_batcher_llm_client.py. To clarify, the lora paths should be in the client side.

[JoJoLev]

@syuoni thanks for the info.
So right now, I have the Lora adapters, I have built a trtllm engine. And have all the proper files.

I am having difficulty enabling this on Sagemaker with triton right now.
Basically, I am wrapping up the files as a tar.gz, and then deploying on triton container.

So, judging by the config.json, I structure my lora path and the rest of the models paths like the following:

Tensorrtllm
|__>1
| |>rank0.engine
| |>config.json
|
|config.pbtxt
|>lora
|>0
| |>lora1 files
|
|>1
|>lora2 files

If I have this files structure on a tar.gz, will I need to add this to config.pbtxt structure?
And then I guess I can call the Lora directory on the client side as an input request?

Hope this clarifies what I am trying to accomplish on this. Thanks!