[Feature Request] More Detailed Logging for In-Flight Batch

[StarrickLiu]

Currently, it seems that the implementation of in-flight batch is closed-source. Is there any way to print some logs for better performance analysis?

For example, we would like to know the batch size for each step in the in-flight batch, as well as the start and end timestamps for each step.

[byshiue]

Have you tried adding --log when you run scripts/launch_triton_server.py? It will return logs including batch size in IFB.

[StarrickLiu]

Have you tried adding --log when you run scripts/launch_triton_server.py? It will return logs including batch size in IFB.

triton_log.txt

I see.

However, this report seems particularly complex. I can't find the information I need in it, such as the BatchSize for each step.

I only found that the BatchSize for each request I sent is 1. Can you point out where the BatchSize for IFB is?

[byshiue]

In line 2386 of your log, you can see info like

I1218 03:35:37.688135 20559 libtensorrtllm.cc:1205] {"Active Request Count":1,"Context Requests":1,"Free KV cache 
blocks":480,"Generation Requests":0,"Iteration Counter":0,"Max KV cache blocks":513,"Max Request Count":64,"MicroBatch 
ID":0,"Runtime CPU Memory Usage":156,"Runtime GPU Memory Usage":3648012,"Runtime Pinned Memory Usage":16,"Scheduled 
Requests":1,"Timestamp":"12-18-2023 03:35:37","Tokens per KV cache block":64,"Total Context Tokens":2048,"Used KV cache
 blocks":1}

[StarrickLiu]

In line 2386 of your log, you can see info like

I1218 03:35:37.688135 20559 libtensorrtllm.cc:1205] {"Active Request Count":1,"Context Requests":1,"Free KV cache 
blocks":480,"Generation Requests":0,"Iteration Counter":0,"Max KV cache blocks":513,"Max Request Count":64,"MicroBatch 
ID":0,"Runtime CPU Memory Usage":156,"Runtime GPU Memory Usage":3648012,"Runtime Pinned Memory Usage":16,"Scheduled 
Requests":1,"Timestamp":"12-18-2023 03:35:37","Tokens per KV cache block":64,"Total Context Tokens":2048,"Used KV cache
 blocks":1}

Do scheduled requests represent the batch size for each step?

[schetlur-nv]

Yes, scheduled requests refers to how many requests were processed in the current iteration. We also print more fine-grained details such as how many requests are in prompt processing (context) phase and how many are in generation phase. This information is returned for every iteration of the IFB generation loop, so you should be able to see fine grained behavior of each iteration.

[pfldy2850]

Hello @schetlur-nv ,

Can you please point me to a configuration variable where I can scale scheduled requests?

There's a bit of a lack of explanation as to what configurations affect to the log, which makes it difficult to understand.
related to NVIDIA/TensorRT-LLM#1012

[lyc728]

"Active Request Count":1

hello, Did you use in-flight batch for acceleration? I shut this down in about the same time as normal triton reasoning. I use the gpt/qwen model
@byshiue @schetlur-nv