Update on the development branch

[kaiyux]

Hi,

The TensorRT-LLM team is pleased to announce that we are pushing an update to the development branch (and the Triton backend) this May 7, 2024.

This update includes:

• Model Support
  • Support CogVLM [Model Requests] Add support for CogVLM #1087
  • Support ByT5
  • Support LLaMA 3
  • Support Arctic (w/ FP8)
  • Support Fuyu
  • Support Persimmon
  • Support Deplot
  • Support Phi-3-Mini with long Rope
• API
  • [BREAKING CHANGE] Add speculative decoding mode to the builder API
• Benchmark
  • A --max_attention_window option is added to gptManagerBenchmark
• Infra
  • Base Docker image for TensorRT-LLM is updated to nvcr.io/nvidia/pytorch:24.03-py3
  • Base Docker image for TensorRT-LLM backend is updated to nvcr.io/nvidia/tritonserver:24.03-py3
  • The dependent TensorRT version is updated to 10.0.1
  • The dependent CUDA version is updated to 12.4.0
  • The dependent PyTorch version is updated to 2.2.2

Thanks,
The TensorRT-LLM Engineering Team