Executor API: How to get throughput

[khayamgondal]

I am looking at benchmarks/python/kv_cache_offload
/benchmark.py example and trying to figure out how can I get throughput. This example gets TTFT from executor.get_latest_iteration_stats(). I looked at get_latest_iteration_stats() definitions but didn't find any information regarding how to get throughput.
Here is what is available via get_latest_iteration_stats() function
'cpu_mem_usage', 'cross_kv_cache_stats', 'gpu_mem_usage', 'inflight_batching_stats', 'iter', 'iter_latency_ms', 'kv_cache_stats', 'max_num_active_requests', 'new_active_requests_queue_latency_ms', 'num_active_requests', 'num_completed_requests', 'num_new_active_requests', 'num_queued_requests', 'pinned_mem_usage', 'static_batching_stats', 'timestamp', 'to_json_str']

[juney-nvidia]

Hi @khayamgondal

The throughput information should be stored in inflight_batching_stats .

Also we are moving to use trtllm-bench to consolidate the performance benchmarking process which I would suggest you to refer.

For any specific question about this kv_cache_offload benchmark script, @SimengLiu-nv may help.

For any question about trtllm-bench, @FrankD412 @kaiyux @jiahanc can help.

Thanks
June

[SimengLiu-nv]

Hi @khayamgondal , the end-to-end throughput statistics are calculated not directly reported. For example, Token Throughput (tokens/sec) = total_output_tokens / total_latency. Request Throughput (req/sec) = total_num_requests / total_latency. For the value of total_latency, you can convert the logged E2E TIME (ms) to second.

[khayamgondal]

Thanks @juney-nvidia how do I usetrtllm.KvCacheConfigwith trtllm-bench?

[kaiyux]

Thanks @juney-nvidia how do I usetrtllm.KvCacheConfigwith trtllm-bench?

@khayamgondal There is a --extra_llm_api_options argument provided by trtllm-bench that allows you to specify any custom configurations following LlmArgs data structure, and that includes kv_cache_config.

Let me know for further questions, thanks.

[FrankD412]

@khayamgondal -- you should be able to specify a KV cache config via an extra options YML file. It would look something like:

kv_cache_config:
  free_gpu_memory_fraction: 0.95
  enable_block_reuse: true

You can find the specific arguments to the KvCacheConfig [here])

TensorRT-LLM/tensorrt_llm/llmapi/llm_args.py

Line 520 in 8bb3eea

class KvCacheConfig(BaseModel, PybindMirror):

). As @kaiyux mentioned, you can pass this in via --extra_llm_api_options (and it isn't limited to just the KV cache. A caveat is that if you modify an entry, none of the configuration set by trtllm-bench will hold so make sure you set everything (even if trtllm-bench does set it).

[khayamgondal]

Thanks @FrankD412 is there a way to dump default values set by trtllm-bench ?

Based on llm_ags.py looks like I need following

kv_cache_config:
    enable_block_reuse: true
    max_tokens: 17800
    max_attention_window: null
    sink_token_length: null
    free_gpu_memory_fraction: 0.9
    host_cache_size: null
    onboard_blocks: true
    cross_kv_cache_fraction: null
    secondary_offload_min_priority: null
    event_buffer_max_size: 0
    enable_partial_reuse: true
    copy_on_partial_reuse: true

[FrankD412]

@khayamgondal -- at the moment there isn't, there is some ongoing effort to make the configuration classes that the LLM API uses pure Python so that we can use Pydantic to serialize/deserialize a configuration YAML, but that's currently still a WIP.

For the KV configuration I think the only thing you'll really need to worry about is the free_gpu_memory_fraction, and if you don't set a kwarg it's just the default.

[khayamgondal]

Thanks @FrankD412 my goal is to play with free_gpu_memory_fraction and host_cache_size to adjust how much KV cache resides on GPU and CPU. I see I can make change at TensorRT-LLM/tensorrt_llm/bench/benchmark/throughput.py to adjust free_gpu_memory_fraction but I don't see any option for host_cache_size in that file. If I can set these two values in the throughput.py script that will be easier for me instead of creating a YAML file for config.

@optgroup.option(
   "--kv_cache_free_gpu_mem_fraction",
   type=float,
   default=.90,
   help="The percentage of memory to use for KV Cache after model load.",
)

[FrankD412]

Got it -- thanks for the feedback @khayamgondal

With the new configuration classes, there may be a way we could add a general CLI option to set properties that don't have a specific CLI option. I'll keep that in mind as we move towards it and see if there's a way to achieve it. We decided to try and limit the options on the CLI to make the benchmark easier to use and only expose the options we thought were the biggest knobs a user would want to tune. For now, the YAML file would be the way to handle it.

[SimengLiu-nv]

@khayamgondal You can try adding the host_cache_size option in https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/bench/dataclasses/configuration.py#L215-L220.