Enable Nvidia's ModelOpt fp8 quantized models

[Edwardf0t1]

Motivation

As discussed in our sync meeting @merrymercy @Ying1123 , we aim to contribute to SGLang by integrating NVIDIA's TensorRT Model Optimizer (ModelOpt) with optimized and quantized models, fostering collaboration to enhance the open-source inference ecosystem.

Modifications

This PR serves as an initial step toward adding support for ModelOpt quantized models in SGLang, starting with FP8 LLaMA 3.1 model inference. A basic test can be executed using the script provided below.

import sglang as sgl

def main():
    prompts = [
        "Hello, my name is",
        "The president of the United States is",
        "The capital of France is",
        "The future of AI is",
    ]
    sampling_params = {"temperature": 0.8, "top_p": 0.95}
    llm = sgl.Engine(model_path="nvidia/Llama-3.1-8B-Instruct-FP8", quantization="modelopt")

    outputs = llm.generate(prompts, sampling_params)
    for prompt, output in zip(prompts, outputs):
        print("===============================")
        print(f"Prompt: {prompt}\nGenerated text: {output['text']}")

if __name__ == "__main__":
    main()

Checklist

• Format your code according to the Contributor Guide.
• Add unit tests as outlined in the Contributor Guide.
• Update documentation as needed, including docstrings or example tutorials.

[zhyncs]

@Edwardf0t1 Please help resolve the conflicts

[Edwardf0t1]

@Edwardf0t1 Please help resolve the conflicts

Done

[merrymercy]

Also, please fix the CI errors.

[merrymercy]

can we make this an optional dependency?

[Edwardf0t1]

IIUC this is already under [project.optional-dependencies]. Also see this comment from @zhyncs :
#2535 (comment)

[Edwardf0t1]

Hi @merrymercy I left a comment in your recently merged PR that I found it could bring issues in my test when run llm = sgl.Engine(model_path="nvidia/Llama-3.1-8B-Instruct-FP8", quantization="modelopt").

[merrymercy]

I see. What is the correct value of GLOO_SOCKET_IFNAME in your environment?

[Edwardf0t1]

I see. What is the correct value of GLOO_SOCKET_IFNAME in your environment?

I can use ens8np0 or enp2s0 interface for GLOO_SOCKET_IFNAME, depending on the system.