Skip to content

Commit 6e6e34e

Browse files
kaiyuxkaiyux
authored
Update TensorRT-LLM backend (triton-inference-server#272)
* Update TensorRT-LLM backend
1 parent f51f50c commit 6e6e34e

35 files changed

+117
-347
lines changed

‎README.md

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
<!--
2-
# Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
33
#
44
# Redistribution and use in source and binary forms, with or without
55
# modification, are permitted provided that the following conditions
@@ -41,20 +41,22 @@ available in the main [server](https://github.com/triton-inference-server/server
4141
repo. If you don't find your answer there you can ask questions on the
4242
[issues page](https://github.com/triton-inference-server/tensorrtllm_backend/issues).
4343

44-
## Building the TensorRT-LLM Backend
44+
## Accessing the TensorRT-LLM Backend
4545

4646
There are several ways to access the TensorRT-LLM Backend.
4747

48-
**Before Triton 23.10 release, please use [Option 3 to build TensorRT-LLM backend via Docker](#option-3-build-via-docker)**
48+
**Before Triton 23.10 release, please use [Option 3 to build TensorRT-LLM backend via Docker](#option-3-build-via-docker).**
4949

50-
### Option 1. Run the Docker Container
50+
### Run the Pre-built Docker Container
5151

5252
Starting with Triton 23.10 release, Triton includes a container with the TensorRT-LLM
5353
Backend and Python Backend. This container should have everything to run a
5454
TensorRT-LLM model. You can find this container on the
5555
[Triton NGC page](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tritonserver).
5656

57-
### Option 2. Build via the build.py Script in Server Repo
57+
### Build the Docker Container
58+
59+
#### Option 1. Build via the `build.py` Script in Server Repo
5860

5961
Starting with Triton 23.10 release, you can follow steps described in the
6062
[Building With Docker](https://github.com/triton-inference-server/server/blob/main/docs/customization_guide/build.md#building-with-docker)
@@ -90,7 +92,7 @@ the TensorRT-LLM backend and Python backend repositories that will be used
9092
to build the container. You can also remove the features or endpoints that you
9193
don't need by removing the corresponding flags.
9294

93-
### Option 3. Build via Docker
95+
#### Option 2. Build via Docker
9496

9597
The version of Triton Server used in this build option can be found in the
9698
[Dockerfile](./dockerfile/Dockerfile.trt_llm_backend).

‎all_models/inflight_batcher_llm/ensemble/config.pbtxt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
1+
# Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
#
33
# Redistribution and use in source and binary forms, with or without
44
# modification, are permitted provided that the following conditions

‎all_models/inflight_batcher_llm/postprocessing/1/model.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
1+
# Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
#
33
# Redistribution and use in source and binary forms, with or without
44
# modification, are permitted provided that the following conditions

‎all_models/inflight_batcher_llm/postprocessing/config.pbtxt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
1+
# Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
#
33
# Redistribution and use in source and binary forms, with or without
44
# modification, are permitted provided that the following conditions

‎all_models/inflight_batcher_llm/preprocessing/1/model.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
1+
# Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
#
33
# Redistribution and use in source and binary forms, with or without
44
# modification, are permitted provided that the following conditions

‎all_models/inflight_batcher_llm/preprocessing/config.pbtxt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
1+
# Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
#
33
# Redistribution and use in source and binary forms, with or without
44
# modification, are permitted provided that the following conditions

‎all_models/inflight_batcher_llm/tensorrt_llm/config.pbtxt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
1+
# Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
#
33
# Redistribution and use in source and binary forms, with or without
44
# modification, are permitted provided that the following conditions

‎all_models/inflight_batcher_llm/tensorrt_llm_bls/1/model.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
1+
# Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
#
33
# Redistribution and use in source and binary forms, with or without
44
# modification, are permitted provided that the following conditions

‎all_models/inflight_batcher_llm/tensorrt_llm_bls/config.pbtxt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
1+
# Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
#
33
# Redistribution and use in source and binary forms, with or without
44
# modification, are permitted provided that the following conditions

‎ci/L0_backend_trtllm/base_metrics_verification_tests.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
#!/usr/bin/python
2-
# Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
33
#
44
# Redistribution and use in source and binary forms, with or without
55
# modification, are permitted provided that the following conditions

‎ci/L0_backend_trtllm/custom_metrics_verification_tests.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
#!/usr/bin/python
2-
# Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
33
#
44
# Redistribution and use in source and binary forms, with or without
55
# modification, are permitted provided that the following conditions

‎ci/L0_backend_trtllm/generate_engines.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
#!/bin/bash
2-
# Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
33
#
44
# Redistribution and use in source and binary forms, with or without
55
# modification, are permitted provided that the following conditions

‎ci/L0_backend_trtllm/test.sh

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
#!/bin/bash
2-
# Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
33
#
44
# Redistribution and use in source and binary forms, with or without
55
# modification, are permitted provided that the following conditions
@@ -39,6 +39,10 @@ CUSTOM_METRICS_VERIFICATION_TEST=custom_metrics_verification_tests.py
3939
CUSTOM_METRICS_VERIFICATION_LOG="custom_metrics_verification.log"
4040
SERVER_PID=0
4141

42+
# Force environment to use python version 3
43+
apt update -q=2 \
44+
&& apt install -y python-is-python3
45+
4246
# Helpers ===============================
4347
function replace_config_tags {
4448
tag_to_replace="${1}"

‎ci/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
<!--
2-
# Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
33
#
44
# Redistribution and use in source and binary forms, with or without
55
# modification, are permitted provided that the following conditions

‎dockerfile/Dockerfile.triton.trt_llm_backend

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
ARG BASE_IMAGE=nvcr.io/nvidia/tritonserver:23.11-py3-min
2+
3+
FROM ${BASE_IMAGE} as base
4+
5+
RUN apt-get update -q=2 && apt-get install -y --no-install-recommends python3-pip
6+
# Remove previous TRT installation
7+
# We didn't remove libnvinfer* here because tritonserver depends on the pre-installed libraries.
8+
RUN apt-get remove -y tensorrt*
9+
RUN pip3 uninstall -y tensorrt
10+
11+
ARG TRT_VER
12+
13+
ENV TRT_VERSION=$TRT_VER \
14+
TRT_VER=$TRT_VER \
15+
CUDA_VER=$CUDA_VERSION \
16+
CUDNN_VER=$CUDNN_VERSION \
17+
NCCL_VER=$NCCL_VERSION \
18+
CUBLAS_VER=$CUBLAS_VERSION
19+
20+
LABEL TRT_VERSION $TRT_VER
21+
22+
RUN echo TRT_VERSION=$TRT_VER \
23+
TRT_VER=$TRT_VER \
24+
CUDA_VER=$CUDA_VERSION \
25+
CUDNN_VER=$CUDNN_VERSION \
26+
NCCL_VER=$NCCL_VERSION \
27+
CUBLAS_VER=$CUBLAS_VERSION
28+
# Download & install internal TRT release
29+
RUN [ "$(uname -m)" != "x86_64" ] && arch="sbsa" || arch="x86_64" \
30+
&& curl -o /tmp/cuda-keyring.deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/$arch/cuda-keyring_1.0-1_all.deb \
31+
&& apt install /tmp/cuda-keyring.deb \
32+
&& rm /tmp/cuda-keyring.deb \
33+
&& apt-get update -q=2
34+
35+
ARG RELEASE_URL_TRT_x86
36+
ARG RELEASE_URL_TRT_ARM
37+
38+
RUN [ "$(uname -m)" != "x86_64" ] && RELEASE_URL_TRT=${RELEASE_URL_TRT_ARM} || RELEASE_URL_TRT=${RELEASE_URL_TRT_x86} \
39+
&& curl -fSL -o /tmp/tensorrt.tar.gz ${RELEASE_URL_TRT} \
40+
&& tar xzvf /tmp/tensorrt.tar.gz -C /usr/local \
41+
&& rm /tmp/tensorrt.tar.gz \
42+
&& find /usr/local -maxdepth 1 -name Tens* -type d -exec ln -s {} /usr/local/tensorrt \;
43+
44+
RUN pip3 install /usr/local/tensorrt/python/tensorrt-*-cp$( python3 -c "import sys; print(str(sys.version_info.major) + str(sys.version_info.minor))" )*
45+
46+
ENV LD_LIBRARY_PATH=/usr/local/tensorrt/lib:${LD_LIBRARY_PATH}
47+
ENV TRT_ROOT=/usr/local/tensorrt

‎inflight_batcher_llm/CMakeLists.txt

Lines changed: 28 additions & 61 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
1+
# Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
#
33
# Redistribution and use in source and binary forms, with or without
44
# modification, are permitted provided that the following conditions are met: *
@@ -28,20 +28,7 @@ set(TRITON_BUILD
2828
OFF
2929
CACHE STRING "Using Triton build process")
3030

31-
if(TRITON_BUILD)
32-
set_ifndef(TRTLLM_DIR ${CMAKE_CURRENT_SOURCE_DIR}/tensorrt_llm)
33-
# Install build time dependencies. This section is executed during cmake
34-
# configure time.
35-
execute_process(
36-
COMMAND bash -x ./tools/environment_setup.sh
37-
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
38-
RESULT_VARIABLE CMD_RESULT)
39-
if(NOT CMD_RESULT EQUAL "0")
40-
message(FATAL_ERROR "Failed to install build time dependencies")
41-
endif()
42-
else()
43-
set_ifndef(TRTLLM_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../tensorrt_llm)
44-
endif()
31+
set_ifndef(TRTLLM_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../tensorrt_llm)
4532

4633
include(${TRTLLM_DIR}/cpp/cmake/modules/find_library_create_target.cmake)
4734

@@ -64,16 +51,6 @@ if(TRITON_ENABLE_METRICS AND NOT TRITON_ENABLE_STATS)
6451
FATAL_ERROR "TRITON_ENABLE_METRICS=ON requires TRITON_ENABLE_STATS=ON")
6552
endif()
6653

67-
# The TRTLLM_BUILD_CONTAINER is used to compile the TRT-LLM libraries that are
68-
# needed for the TRT-LLM backend. The TRTLLM_BUILD_CONTAINER is launched
69-
# separately, and the artifacts will be copied back to the backend installation
70-
# directory.
71-
if(TRITON_BUILD)
72-
set(TRTLLM_BUILD_CONTAINER
73-
""
74-
CACHE STRING "Base image for building TRT-LLM")
75-
endif()
76-
7754
set(TRITON_COMMON_REPO_TAG
7855
"main"
7956
CACHE STRING "Tag for triton-inference-server/common repo")
@@ -116,31 +93,6 @@ FetchContent_Declare(
11693
GIT_SHALLOW ON)
11794
FetchContent_MakeAvailable(repo-common repo-core repo-backend)
11895

119-
# Compile TRT-LLM
120-
if(TRITON_BUILD)
121-
set(TRITON_TRTLLM_DOCKER_NAME "tritonserver-trtllm")
122-
add_custom_command(
123-
OUTPUT tensorrt_llm_build
124-
COMMENT "Building TensorRT-LLM"
125-
COMMAND
126-
cd ${CMAKE_CURRENT_SOURCE_DIR} && python3 tools/gen_trtllm_dockerfile.py
127-
--trtllm-build-config="${CMAKE_BUILD_TYPE}"
128-
--trtllm-base-image="${TRTLLM_BUILD_CONTAINER}" --output=Dockerfile.trtllm
129-
COMMAND
130-
cd ${CMAKE_CURRENT_SOURCE_DIR} && docker build
131-
--cache-from=${TRITON_TRTLLM_DOCKER_NAME}
132-
--cache-from=${TRITON_TRTLLM_DOCKER_NAME}_cache0
133-
--cache-from=${TRITON_TRTLLM_DOCKER_NAME}_cache1 -t
134-
${TRITON_TRTLLM_DOCKER_NAME} -f ./Dockerfile.trtllm .
135-
COMMAND docker rm trtllm_build || echo 'error ignored...' || true
136-
COMMAND docker create --name trtllm_build ${TRITON_TRTLLM_DOCKER_NAME}
137-
COMMAND cd ${CMAKE_CURRENT_SOURCE_DIR} && rm -fr tensorrt_llm
138-
COMMAND cd ${CMAKE_CURRENT_SOURCE_DIR} && docker cp
139-
trtllm_build:/app/tensorrt_llm tensorrt_llm
140-
COMMAND docker cp trtllm_build:/opt/trtllm_lib trtllm_build
141-
COMMAND docker rm trtllm_build)
142-
endif()
143-
14496
#
14597
# The backend must be built into a shared library. Use an ldscript to hide all
14698
# symbols except for the TRITONBACKEND API.
@@ -153,11 +105,6 @@ set(SRCS src/libtensorrtllm.cc src/work_item.cc src/work_items_queue.cc
153105

154106
add_library(triton-tensorrt-llm-backend SHARED ${SRCS})
155107

156-
if(TRITON_BUILD)
157-
add_custom_target(trtllm_target DEPENDS tensorrt_llm_build)
158-
add_dependencies(triton-tensorrt-llm-backend trtllm_target)
159-
endif()
160-
161108
add_library(TritonTensorRTLLMBackend::triton-tensorrt-llm-backend ALIAS
162109
triton-tensorrt-llm-backend)
163110

@@ -352,10 +299,25 @@ if(TRITON_ENABLE_METRICS)
352299
endif()
353300

354301
if(TRITON_BUILD)
355-
add_dependencies(tensorrt_llm trtllm_target)
356-
add_dependencies(tensorrt_llm_batch_manager trtllm_target)
357-
add_dependencies(nvinfer_plugin_tensorrt_llm trtllm_target)
358-
endif()
302+
303+
if(CMAKE_HOST_SYSTEM_PROCESSOR STREQUAL "x86_64")
304+
execute_process(
305+
WORKING_DIRECTORY ${TRTLLM_DIR}
306+
COMMAND bash -x docker/common/install_pytorch.sh pypi COMMAND_ECHO STDOUT
307+
COMMAND_ERROR_IS_FATAL ANY)
308+
else()
309+
execute_process(
310+
WORKING_DIRECTORY ${TRTLLM_DIR}
311+
COMMAND bash -x docker/common/install_pytorch.sh src_non_cxx11_abi
312+
COMMAND_ECHO STDOUT COMMAND_ERROR_IS_FATAL ANY)
313+
endif() # CMAKE_HOST_SYSTEM_PROCESSOR
314+
315+
execute_process(
316+
WORKING_DIRECTORY ${TRTLLM_DIR}
317+
COMMAND python3 scripts/build_wheel.py --trt_root /usr/local/tensorrt
318+
COMMAND_ECHO STDOUT COMMAND_ERROR_IS_FATAL ANY)
319+
320+
endif() # TRITON_BUILD
359321

360322
target_link_libraries(
361323
triton-tensorrt-llm-backend
@@ -407,9 +369,14 @@ install(
407369
RUNTIME DESTINATION ${CMAKE_INSTALL_PREFIX}/backends/tensorrtllm)
408370

409371
if(TRITON_BUILD)
410-
install(DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/trtllm_build/
372+
file(
373+
GLOB
374+
LIBINFER_PLUGIN_TENSORRT_LLM
375+
"${TRTLLM_DIR}/cpp/build/tensorrt_llm/plugins/libnvinfer_plugin_tensorrt_llm.so*"
376+
FOLLOW_SYMLINKS)
377+
install(FILES ${LIBINFER_PLUGIN_TENSORRT_LLM}
411378
DESTINATION ${CMAKE_INSTALL_PREFIX}/backends/tensorrtllm)
412-
endif()
379+
endif() # TRITON_BUILD
413380

414381
install(
415382
EXPORT triton-tensorrt-llm-backend-targets

‎inflight_batcher_llm/client/inflight_batcher_llm_client.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
#!/usr/bin/env python
2-
# Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
33
#
44
# Redistribution and use in source and binary forms, with or without
55
# modification, are permitted provided that the following conditions

‎inflight_batcher_llm/cmake/TritonTensorRTLLMBackendConfig.cmake.in

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
1+
# Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
#
33
# Redistribution and use in source and binary forms, with or without
44
# modification, are permitted provided that the following conditions

‎inflight_batcher_llm/src/custom_metrics_reporter/custom_metrics_reporter.cc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
// Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
1+
// Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
//
33
// Redistribution and use in source and binary forms, with or without
44
// modification, are permitted provided that the following conditions

‎inflight_batcher_llm/src/custom_metrics_reporter/custom_metrics_reporter.h

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
// Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
1+
// Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
//
33
// Redistribution and use in source and binary forms, with or without
44
// modification, are permitted provided that the following conditions

‎inflight_batcher_llm/src/libtensorrtllm.cc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
// Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
1+
// Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
//
33
// Redistribution and use in source and binary forms, with or without
44
// modification, are permitted provided that the following conditions

‎inflight_batcher_llm/src/model_instance_state.cc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
// Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
1+
// Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
//
33
// Redistribution and use in source and binary forms, with or without
44
// modification, are permitted provided that the following conditions

‎inflight_batcher_llm/src/model_instance_state.h

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
// Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
1+
// Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
//
33
// Redistribution and use in source and binary forms, with or without
44
// modification, are permitted provided that the following conditions

‎inflight_batcher_llm/src/model_state.cc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
// Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
1+
// Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
//
33
// Redistribution and use in source and binary forms, with or without
44
// modification, are permitted provided that the following conditions

‎inflight_batcher_llm/src/model_state.h

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
// Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
1+
// Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
//
33
// Redistribution and use in source and binary forms, with or without
44
// modification, are permitted provided that the following conditions

‎inflight_batcher_llm/src/utils.cc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
// Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
1+
// Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
//
33
// Redistribution and use in source and binary forms, with or without
44
// modification, are permitted provided that the following conditions

‎inflight_batcher_llm/src/utils.h

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
// Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
1+
// Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
//
33
// Redistribution and use in source and binary forms, with or without
44
// modification, are permitted provided that the following conditions

‎inflight_batcher_llm/src/work_item.cc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
// Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
1+
// Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
//
33
// Redistribution and use in source and binary forms, with or without
44
// modification, are permitted provided that the following conditions

‎inflight_batcher_llm/src/work_item.h

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
// Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
1+
// Copyright 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
//
33
// Redistribution and use in source and binary forms, with or without
44
// modification, are permitted provided that the following conditions

0 commit comments

Comments
 (0)