TRT-LLM loading mechanism tool #3398

apbose · 2025-02-14T17:58:18Z

TRT-LLM download utility

narendasan · 2025-02-25T22:13:19Z

py/torch_tensorrt/dynamo/conversion/converter_utils.py

-                f"Ensure the path is correct and the library is compatible",
-                exc_info=e_os_error,
+        else:
+            py_version = f"cp{sys.version_info.major}{sys.version_info.minor}"


Why do we restrict to cp310 and cp312, It shouldnt matter if we are pulling the whl and unzipping ourselves

https://pypi.nvidia.com/tensorrt-llm/ In this since I see the tags for only cp310 and cp312 I added the check

py/torch_tensorrt/dynamo/conversion/converter_utils.py

github-actions

There are some changes that do not conform to Python style guidelines:

--- /home/runner/work/TensorRT/TensorRT/tests/py/dynamo/conversion/test_nccl_ops.py	2025-02-27 20:03:00.014038+00:00
+++ /home/runner/work/TensorRT/TensorRT/tests/py/dynamo/conversion/test_nccl_ops.py	2025-02-27 20:03:24.885031+00:00
@@ -22,11 +22,11 @@

from .harness import DispatchTestCase


class TestGatherNcclOpsConverter(DispatchTestCase):
-    @parameterized.expand([(8)])
+    @parameterized.expand([8])
    def test_nccl_ops(self, linear_layer_dim):
        class DistributedGatherModel(nn.Module):
            def __init__(self, input_dim):
                super().__init__()
                self.fc = torch.nn.Linear(input_dim, input_dim)

github-actions

There are some changes that do not conform to Python style guidelines:

--- /home/runner/work/TensorRT/TensorRT/tests/py/dynamo/conversion/test_nccl_ops.py	2025-02-27 20:05:38.023287+00:00
+++ /home/runner/work/TensorRT/TensorRT/tests/py/dynamo/conversion/test_nccl_ops.py	2025-02-27 20:06:02.662188+00:00
@@ -22,11 +22,11 @@

from .harness import DispatchTestCase


class TestGatherNcclOpsConverter(DispatchTestCase):
-    @parameterized.expand([(8)])
+    @parameterized.expand([8])
    def test_nccl_ops(self, linear_layer_dim):
        class DistributedGatherModel(nn.Module):
            def __init__(self, input_dim):
                super().__init__()
                self.fc = torch.nn.Linear(input_dim, input_dim)

github-actions

There are some changes that do not conform to Python style guidelines:

--- /home/runner/work/TensorRT/TensorRT/tests/py/dynamo/conversion/test_nccl_ops.py	2025-02-27 20:05:54.405311+00:00
+++ /home/runner/work/TensorRT/TensorRT/tests/py/dynamo/conversion/test_nccl_ops.py	2025-02-27 20:06:21.454993+00:00
@@ -22,11 +22,11 @@

from .harness import DispatchTestCase


class TestGatherNcclOpsConverter(DispatchTestCase):
-    @parameterized.expand([(8)])
+    @parameterized.expand([8])
    def test_nccl_ops(self, linear_layer_dim):
        class DistributedGatherModel(nn.Module):
            def __init__(self, input_dim):
                super().__init__()
                self.fc = torch.nn.Linear(input_dim, input_dim)

github-actions

There are some changes that do not conform to Python style guidelines:

--- /home/runner/work/TensorRT/TensorRT/tests/py/dynamo/conversion/test_nccl_ops.py	2025-04-15 19:58:05.267724+00:00
+++ /home/runner/work/TensorRT/TensorRT/tests/py/dynamo/conversion/test_nccl_ops.py	2025-04-15 19:58:36.145897+00:00
@@ -22,11 +22,11 @@

from .harness import DispatchTestCase


class TestGatherNcclOpsConverter(DispatchTestCase):
-    @parameterized.expand([(8)])
+    @parameterized.expand([8])
    def test_nccl_ops(self, linear_layer_dim):
        class DistributedGatherModel(nn.Module):
            def __init__(self, input_dim):
                super().__init__()
                self.fc = torch.nn.Linear(input_dim, input_dim)

github-actions

There are some changes that do not conform to Python style guidelines:

--- /home/runner/work/TensorRT/TensorRT/tests/py/dynamo/conversion/test_nccl_ops.py	2025-04-15 21:00:13.719714+00:00
+++ /home/runner/work/TensorRT/TensorRT/tests/py/dynamo/conversion/test_nccl_ops.py	2025-04-15 21:00:40.093669+00:00
@@ -22,11 +22,11 @@

from .harness import DispatchTestCase


class TestGatherNcclOpsConverter(DispatchTestCase):
-    @parameterized.expand([(8)])
+    @parameterized.expand([8])
    def test_nccl_ops(self, linear_layer_dim):
        class DistributedGatherModel(nn.Module):
            def __init__(self, input_dim):
                super().__init__()
                self.fc = torch.nn.Linear(input_dim, input_dim)

py/torch_tensorrt/dynamo/_compiler.py

py/torch_tensorrt/dynamo/utils.py

narendasan

Use a temp directory to save the wheel and unzipped wheel

py/torch_tensorrt/dynamo/utils.py

tests/py/dynamo/distributed/test_nccl_ops.py

py/torch_tensorrt/dynamo/utils.py

narendasan

Seems like its close are there any tests for the downloader? like verifying the correct file is downloaded and available?

apbose · 2025-07-07T19:05:08Z

The test tests/py/dynamo/distributed/test_nccl_ops.py should verify it. The USE_TRTLLM_PLUGINS=1 pytest distributed/test_nccl_ops.py is the command. It is not part of CI though

…ing in dynamo.compile TRT-LLM installation utilities and adding test cases adding the option in _compiler.py changes in the TRT-LLM loading tool- removing install_wget, install_unzip, install_mpi Further changes in error logging of the TRT-LLM installation tool moving the load_tensorrt_llm to dynamo/utils.py correcting misprint for TRT LLM load Using python lib for download to make it platform agnostic dll file path update for windows correcting the non critical lint error Including version in versions.txt

…action, variable for py_version

apbose self-assigned this Feb 14, 2025

facebook-github-bot added the cla signed label Feb 14, 2025

apbose marked this pull request as draft February 14, 2025 17:58

github-actions bot added component: conversion Issues re: Conversion stage component: api [Python] Issues re: Python API component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths labels Feb 14, 2025

github-actions bot requested a review from peri044 February 14, 2025 17:58

apbose force-pushed the nccl_ops_trt_llm_installation branch from 57dbb3f to 3e38e87 Compare February 25, 2025 14:34

narendasan reviewed Feb 25, 2025

View reviewed changes

py/torch_tensorrt/dynamo/conversion/converter_utils.py Outdated Show resolved Hide resolved

narendasan reviewed Feb 25, 2025

View reviewed changes

py/torch_tensorrt/dynamo/conversion/converter_utils.py Show resolved Hide resolved

github-actions bot added the component: tests Issues re: Tests label Feb 27, 2025

github-actions bot requested changes Feb 27, 2025

View reviewed changes

apbose force-pushed the nccl_ops_trt_llm_installation branch from 9ba407b to 5f3fdac Compare February 27, 2025 20:05

apbose marked this pull request as ready for review February 27, 2025 20:05

github-actions bot requested changes Feb 27, 2025

View reviewed changes

apbose force-pushed the nccl_ops_trt_llm_installation branch from 5f3fdac to b66350e Compare April 15, 2025 19:57

github-actions bot requested changes Apr 15, 2025

View reviewed changes

apbose changed the title ~~change in TRT-LLM loading mechanism and exposing aot_joint_export in _compiler.py~~ TRT-LLM loading mechanism tool Apr 15, 2025

github-actions bot requested changes Apr 15, 2025

View reviewed changes

github-actions bot removed the component: tests Issues re: Tests label Apr 17, 2025

apbose force-pushed the nccl_ops_trt_llm_installation branch 2 times, most recently from 6e893ed to 77f2145 Compare April 18, 2025 00:51

narendasan reviewed Apr 25, 2025

View reviewed changes

py/torch_tensorrt/dynamo/_compiler.py Outdated Show resolved Hide resolved

py/torch_tensorrt/dynamo/utils.py Outdated Show resolved Hide resolved

apbose force-pushed the nccl_ops_trt_llm_installation branch from f30acb7 to 9c238ae Compare April 29, 2025 21:33

narendasan reviewed May 2, 2025

View reviewed changes

py/torch_tensorrt/dynamo/utils.py Outdated Show resolved Hide resolved

apbose force-pushed the nccl_ops_trt_llm_installation branch 2 times, most recently from 89d621d to 27aa2f2 Compare May 2, 2025 02:36

apbose commented May 2, 2025

View reviewed changes

py/torch_tensorrt/dynamo/utils.py Outdated Show resolved Hide resolved

narendasan reviewed Jun 13, 2025

View reviewed changes

py/torch_tensorrt/dynamo/utils.py Outdated Show resolved Hide resolved

narendasan reviewed Jun 13, 2025

View reviewed changes

py/torch_tensorrt/dynamo/utils.py Outdated Show resolved Hide resolved

narendasan reviewed Jun 13, 2025

View reviewed changes

py/torch_tensorrt/dynamo/utils.py Outdated Show resolved Hide resolved

narendasan reviewed Jun 13, 2025

View reviewed changes

py/torch_tensorrt/dynamo/utils.py Outdated Show resolved Hide resolved

narendasan reviewed Jun 13, 2025

View reviewed changes

apbose force-pushed the nccl_ops_trt_llm_installation branch from 37d4e90 to e8bc3a4 Compare June 13, 2025 21:54

apbose force-pushed the nccl_ops_trt_llm_installation branch 2 times, most recently from a0cecf4 to 4193ca2 Compare July 1, 2025 09:17

github-actions bot added the component: tests Issues re: Tests label Jul 1, 2025

apbose force-pushed the nccl_ops_trt_llm_installation branch 3 times, most recently from 4589c76 to 1e2148d Compare July 1, 2025 18:23

narendasan reviewed Jul 1, 2025

View reviewed changes

py/torch_tensorrt/dynamo/utils.py Outdated Show resolved Hide resolved

apbose force-pushed the nccl_ops_trt_llm_installation branch from 42d4862 to 9cb3cab Compare July 1, 2025 22:35

apbose commented Jul 1, 2025

View reviewed changes

tests/py/dynamo/distributed/test_nccl_ops.py Outdated Show resolved Hide resolved

narendasan reviewed Jul 2, 2025

View reviewed changes

py/torch_tensorrt/dynamo/utils.py Show resolved Hide resolved

apbose force-pushed the nccl_ops_trt_llm_installation branch 2 times, most recently from dbfd7ee to 15d681a Compare July 4, 2025 00:40

narendasan reviewed Jul 7, 2025

View reviewed changes

apbose added 9 commits July 7, 2025 14:20

linting error fixes and rebase fix

bda179a

removing Platform enum from converter_utils.py

aa0b7ad

Addressing review comments- tmp dir for wheel download and wheel extr…

da5eb7c

…action, variable for py_version

checks for windows where NCCL backend is not supported

f4bd5c9

adding checks for windows and jetson devices

ad8d4fe

Keeping the extracted and deleting download file, restructuring test

46faa4d

modifying the error warning of missing libmpi libs

2374caf

removing the redundant initializations

340182b

apbose force-pushed the nccl_ops_trt_llm_installation branch from c8b8337 to 340182b Compare July 7, 2025 21:20

TRT-LLM loading mechanism tool #3398

Are you sure you want to change the base?

TRT-LLM loading mechanism tool #3398

Uh oh!

Conversation

apbose commented Feb 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

narendasan Feb 25, 2025

Choose a reason for hiding this comment

Uh oh!

apbose Feb 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

narendasan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

narendasan left a comment

Choose a reason for hiding this comment

Uh oh!

apbose commented Jul 7, 2025

Uh oh!

Uh oh!

apbose commented Feb 14, 2025 •

edited

Loading