SAFETENSORS and OpenAI style endpoint

[RonanKMcGovern]

System Info

I have searched the repo here and the main server repo but don't see any information on either a) support for Safetensors (many models are saved that way on HF) and also b) whether there is an openai style endpoint that can be hit?

Seems like this library is getting better speed over vLLM and TGI for DBRX (and in general) so I'd be keen to test this out and see if I can get a docker image running.

Who can help?

@juney-nvidia

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

See docs

Expected behavior

It would be helpful to have info on endpoint style and also supported file types (specifically safetensors) in the docs.

actual behavior

NA

additional notes

NA

[MrD005]

@RonanKMcGovern For your first point.
TensorRT LLM support Different LLM architecture and you can use TensorRT-LLM and its example code to convert safetensors model to tensorRT engine file which can further be used for deployment.

[ishandhanani]

@RonanKMcGovern - I don't think triton has native support for an OAI compatible style endpoint yet.

My current flow is converting a HF model via trt-llm and then feeding the engine into triton

[RonanKMcGovern]

Thanks

…

On Tue, Apr 2, 2024 at 7:50 PM ishandhanani ***@***.***> wrote: @RonanKMcGovern <https://github.com/RonanKMcGovern> - I don't think triton has native support for an OAI compatible style endpoint yet. My current flow is converting a HF model via trt-llm and then feeding the engine into triton — Reply to this email directly, view it on GitHub <#388 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASVG6CXPZAZEFDYONB6VGRDY3L4Y7AVCNFSM6AAAAABFRLLIM6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZSHAYTKMJUGI> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[MrD005]

@RonanKMcGovern you can use https://github.com/npuichigo/openai_trtllm , it is a wrapper to create openai compatible api for tensorRT-LLM

[RonanKMcGovern]

Thanks!

…

On Mon, Jul 29, 2024 at 9:22 AM Dev Goel ***@***.***> wrote: @RonanKMcGovern <https://github.com/RonanKMcGovern> you can use https://github.com/npuichigo/openai_trtllm , it is a wrapper to create openai compatible api for tensorRT-LLM — Reply to this email directly, view it on GitHub <#388 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASVG6CXE34Z3523UBOGNGGLZOX3S7AVCNFSM6AAAAABFRLLIM6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJVGMYDKMRYGI> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>