test: Increase E2E load test token counts to 1500/1500 - #550
Closed
clubanderson wants to merge 1 commit into
Closed
Conversation
Increase input and output token counts from 64 to 1500 to make E2E load tests more realistic. This change: - Makes requests take longer, building queue faster - Increases KV cache utilization to realistic levels - Gives EPP more time to observe and react to queue imbalances - Better simulates production LLM inference workloads Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR increases the token counts used in E2E load tests to better simulate production workloads and test EPP routing behavior under sustained load. The change updates both input and output token constants from 64 to 1500 tokens.
Key Changes:
- Updated token configuration to use 1500 tokens (input and output) instead of 64
- Enables more realistic load testing scenarios with longer-running requests
| outputTokens = 64 | ||
| inputTokens = 1500 | ||
| outputTokens = 1500 | ||
| maxExecutionTimeSec = 600 |
There was a problem hiding this comment.
With token counts increasing from 64 to 1500 (23x increase), the current 600-second timeout may be insufficient for tests to complete, especially under load conditions. Consider increasing maxExecutionTimeSec proportionally to account for longer inference times with 1500 tokens.
Suggested change
| maxExecutionTimeSec = 600 | |
| maxExecutionTimeSec = 14400 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes
inputTokens: 64 → 1500outputTokens: 64 → 1500Rationale
With 64 tokens, requests complete very quickly, making it harder to:
1500 tokens better simulates production inference workloads.
Test plan
🤖 Generated with Claude Code