Skip to content

fix: correct batchsize dimension hint for tasks in enc_dec models #4247

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

fix: correct batchsize dimension hint for tasks in enc_dec models #4247

wants to merge 2 commits into from

Conversation

ducviet00
Copy link

@ducviet00 ducviet00 commented May 13, 2025 •
edited
Loading

Description

This PR fixes error raising when use promp tuning with --remove_input_padding disable

With ('batch_size', bs_range), you will always get this below error:

AssertionError: Expecting all the dimensions in the dim_range has same number of profiles, tensor 'tasks' got dim_range=OrderedDict({'batch_size': [1, 16, 32], 'broadcast_dim': [1]})

This happens when I tried to build encoder decoder model for a multimodal which requires prompt tuning.

@ducviet00
Copy link
Author

/bot run

@ducviet00 ducviet00 force-pushed the fix_remove_input_padding_multimodal branch 4 times, most recently from cd35fff to 9654dd8 Compare May 13, 2025 13:54
@ducviet00
Copy link
Author

@kaiyux Could you please take a look?

@ducviet00 ducviet00 changed the title fix: correct batchsize for tasks in enc_dec models fix: correct batchsize dimension hint for tasks in enc_dec models May 15, 2025
@ducviet00
Copy link
Author

Hi @VALLIS-NERIA @byshiue
Please take a look 🙌

@VALLIS-NERIA
Copy link
Collaborator

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #5270 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #5270 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #3850 completed with status: 'FAILURE'

@ducviet00
Copy link
Author

@VALLIS-NERIA may I have the failed test?

@ducviet00 ducviet00 force-pushed the fix_remove_input_padding_multimodal branch from efaa1d6 to 0e41d11 Compare June 5, 2025 01:16
@poweiw poweiw added Community want to contribute PRs initiated from Community triaged Issue has been triaged by maintainers Generic Runtime General operational aspects of TRTLLM execution not in other categories. labels Jun 5, 2025
@VALLIS-NERIA
Copy link
Collaborator

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #8728 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #8728 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #6331 completed with status: 'ABORTED'

@kaiyux kaiyux requested a review from symphonylyh June 27, 2025 05:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@symphonylyh symphonylyh Awaiting requested review from symphonylyh

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
Community want to contribute PRs initiated from Community Generic Runtime General operational aspects of TRTLLM execution not in other categories. triaged Issue has been triaged by maintainers
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@ducviet00 @VALLIS-NERIA @tensorrt-cicd @poweiw