Poor performance after FP8 Quantization for Llama 3.1 on PyTorch backend

[geaned]

System Info

• CPU architecture: x86_64
• CPU memory size: ~1TB
• GPU properties:
  • GPU name: NVIDIA H100
  • GPU memory size: 80GB
• Libraries:
  • Using docker image nvcr.io/nvidia/tritonserver:25.04-pyt-python-py3 with the following libraries installed afterwards:
    • tensorrt-llm==0.20.0rc3
    • torch==2.7.0
    • transformers==4.51.3
• NVIDIA driver version: 535.161.08
• CUDA version: 12.9
• OS: 24.04.2 LTS

Who can help?

@Tracin @kaiyux

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

I am currently trying to estimate potential throughput increase with implementing EAGLE-3 and noticed that the quantized model actually performs worse that non-quantized if speculative decoding is not used, and on the other side quantized model performs better with EAGLE-3 enabled.

Steps to reproduce the behaviour:

1. Obtain Llama 3.1 8B Instruct and the FP8 its quantized version from corresponding repos (meta-llama/Llama-3.1-8B-Instruct and nvidia/Llama-3.1-8B-Instruct-FP8). Also obtain pretrained EAGLE-3 weights for Llama 3.1 from https://huggingface.co/lmsys/sglang-EAGLE3-LLaMA3.1-Instruct-8B
2. Obtain the dataset targeted to generation speed measurement from https://pastebin.com/42mig2um
   2.1. Input tokens are built using the model.tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True) command
3. Launch a concurrent benchmark with 16 & 128 connections for approximately 10 minutes for each scenario

My result for 16 connections

Llama-3.1-8B-Instruct
====================================
First Token Latency P90 =    29ms

First Token Latency P95 =    29ms

First Token Latency P99 =    30ms

Total Response Latency AVG = 10.34s

Requests per second =        1.53

Output tokens per second =   1149.48

Llama-3.1-8B-Instruct-FP8
====================================
First Token Latency P90 =    32ms

First Token Latency P95 =    32ms

First Token Latency P99 =    37ms

Total Response Latency AVG = 11.25s

Requests per second =        1.41
Output tokens per second =   1056.64

Llama-3.1-8B-Instruct w EAGLE3
====================================
First Token Latency P90 =    40ms

First Token Latency P95 =    41ms

First Token Latency P99 =    46ms

Total Response Latency AVG = 4.81s

Requests per second =        3.32
Output tokens per second =   2538.31

Llama-3.1-8B-Instruct-FP8 w EAGLE3
====================================
First Token Latency P90 =    44ms

First Token Latency P95 =    45ms

First Token Latency P99 =    52ms

Total Response Latency AVG = 5.15s

Requests per second =        3.11

Output tokens per second =   2303.76

My results for 128 connections

Llama-3.1-8B-Instruct
====================================
First Token Latency P90 =    52ms

First Token Latency P95 =    65ms

First Token Latency P99 =    144ms

Total Response Latency AVG = 12.49s

Requests per second =        10.01

Output tokens per second =   7628.94

Llama-3.1-8B-Instruct-FP8
====================================
First Token Latency P90 =    56ms

First Token Latency P95 =    59ms

First Token Latency P99 =    139ms

Total Response Latency AVG = 13.18s

Requests per second =        9.49

Output tokens per second =   6915.62

Llama-3.1-8B-Instruct w EAGLE3
====================================
First Token Latency P90 =    98ms

First Token Latency P95 =    99ms

First Token Latency P99 =    130ms

Total Response Latency AVG = 11.53s

Requests per second =        11.06

Output tokens per second =   8493.62

Llama-3.1-8B-Instruct-FP8 w EAGLE3
====================================
First Token Latency P90 =    87ms

First Token Latency P95 =    90ms

First Token Latency P99 =    127ms

Total Response Latency AVG = 10.03s

Requests per second =        12.67

Output tokens per second =   9324.91

Could you please clarify any possible reasons for such behaviour? Can this be considered a bug with inference of quantized models on PyTorch backend?

Moreover, do you have any ideas for what reason my GPU seems to be underutilized when I use the PyTorch backend? For every setup from the list above my GPU utilization did not go beyond 70%, but when the Triton Backend (https://github.com/triton-inference-server/backend) is used, the GPU is utilized fully.

Expected behavior

Full GPU utilization when using PyTorch backend, and FP8 model performing better than FP16 model (with and without EAGLE, correspondingly)

actual behavior

GPU underutilized (<70%), and FP8 model performs worse than FP16 in all scenarios except the scenario with the largest load (128 concurrent connections + EAGLE3)

additional notes

[geaned]

Just in case, the model is initialized with the following parameters:

model = LLM(
    model=<MODEL_PATH>,
    pytorch_backend_config=PyTorchConfig(
        disable_overlap_scheduler=True,
        use_kv_cache=True,
        kv_cache_dtype="auto",
        attn_backend="TRTLLM",
        load_format="auto",
        print_iter_log=False,
        enable_iter_perf_stats=False,
        torch_compile_enabled=False,
        use_cuda_graph=False,
        enable_trtllm_sampler=False
    ),
    kv_cache_config=KvCacheConfig(
        enable_block_reuse=False,
        enable_partial_reuse=True,
        free_gpu_memory_fraction=0.98
    ),
    scheduler_config=SchedulerConfig(
        capacity_scheduler_policy="MAX_UTILIZATION",
        context_chunking_policy="FIRST_COME_FIRST_SERVED"
    ),
    # >>> When EAGLE is used
    speculative_config=EagleDecodingConfig(
        pytorch_eagle_weights_path=<EAGLE_PATH>,
        max_draft_len=4
    ),
    # <<<
    max_seq_len=32768,
    max_batch_size=128,
    max_num_tokens=32768,
    batching_type="INFLIGHT",
    enable_chunked_prefill=True,
    enable_lora=False,
    tensor_parallel_size=1,
    pipeline_parallel_size=1,
    enable_attention_dp=False
)

And tokens are sampled greedily.

[geaned]

I also checked whether the request per second and output tokens per second drop on Triton Inference Server (chose version 0.19.0 since it is easy to setup via nvcr.io/nvidia/tritonserver:25.05-trtllm-python-py3 docker image), and I believe it is not the case

My results for 16 connections on Triton Inference Server

Llama-3.1-8B-Instruct
====================================
First Token Latency P90 =    26ms

First Token Latency P95 =    27ms

First Token Latency P99 =    28ms

Total Response Latency AVG = 8.95s

Requests per second =        2.00
Output tokens per second =   1533.78

Llama-3.1-8B-Instruct-FP8
====================================
First Token Latency P90 =    24ms

First Token Latency P95 =    25ms

First Token Latency P99 =    26ms

Total Response Latency AVG = 7.29s

Requests per second =        2.18
Output tokens per second =   1669.17

My results for 128 connections on Triton Inference Server

Llama-3.1-8B-Instruct
====================================
First Token Latency P90 =    38ms

First Token Latency P95 =    41ms

First Token Latency P99 =    45ms

Total Response Latency AVG = 9.98s

Requests per second =        12.75
Output tokens per second =   9887.44

Llama-3.1-8B-Instruct-FP8
====================================
First Token Latency P90 =    28ms

First Token Latency P95 =    29ms

First Token Latency P99 =    30ms

Total Response Latency AVG = 9.08s

Requests per second =        14.03
Output tokens per second =   10590.74

[VALLIS-NERIA]

use_cuda_graph=False

Perhaps the CPU runtime is bonding. Can you try turning CUDA graph on?