Skip to content

[CB] bug fix: account for prefill token #320

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jul 21, 2025
Merged

[CB] bug fix: account for prefill token #320

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
58b6de8
account for prefill token
yannicks1 Jul 17, 2025
07ec941
adapt script long_context.py
yannicks1 Jul 17, 2025
283d752
address feedback: renaming var
yannicks1 Jul 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions examples/offline_inference/long_context.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@
trunc = args.trunc_print_len

max_num_seqs = args.max_num_seqs # defines the max batch size
assert args.max_prompt_len < args.max_model_len
assert args.max_prompt_len <= args.max_model_len

if platform.machine() == "arm64":
print("Detected arm64 running environment. "
Expand Down Expand Up @@ -122,7 +122,7 @@ def round_up(t):


tokens_to_generate = [
args.max_model_len - round_up(plen) for plen in prompt_lens
args.max_model_len + 1 - round_up(plen) for plen in prompt_lens
Copy link
Collaborator

@sducouedic sducouedic Jul 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I cannot help but read plen as if it was one word, can we rename that?

Suggested change
args.max_model_len + 1 - round_up(plen) for plen in prompt_lens
args.max_model_len + 1 - round_up(p_len) for p_len in prompt_lens

or

Suggested change
args.max_model_len + 1 - round_up(plen) for plen in prompt_lens
args.max_model_len + 1 - round_up(prompt_len) for prompt_lenin prompt_lens

Copy link
Collaborator

@joerunde joerunde Jul 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for prompt_len, we don't need to save the bytes

Copy link
Collaborator

@joerunde joerunde Jul 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prompt_lenin

]

sampling_params = [
Expand Down
3 changes: 2 additions & 1 deletion vllm_spyre/platform.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -284,7 +284,8 @@ def validate_request(
# ceil division to pad to next block boundary
prompt_padding_len = math.ceil(
prompt_len / cls._block_size) * cls._block_size
if (prompt_padding_len + max_tokens
# we have to account for the token generated during prefill (-1)
if (prompt_padding_len + max_tokens - 1
> cls._config.scheduler_config.max_model_len):
raise ValueError(
"Could not add request: prompt length is "
Expand Down