What each type of buffer is doing?

[alxmamaev]

I was trying to modify TRT LLM code to make a specific thing for my model, that’s is not implemented, and was quite confused with amount of buffers that’s existed can you please explain the structure of them?

• Runtime buffers are using for storing values for TRT engine inputs and outputs
• Decoder buffers, if I get right decoder means not actual decoder, rather than sampler that sample discrete tokens from models logins and doing all beam search logic which is needed
• Decoder inputs and outputs - still does not understand why do we need them, I we already have decoder buffers
• Encoder buffers - why do we need them as well if encoder is running in separate model instance which has their own runtime buffers
• req slots - what is that?
• why executor instance have multiple buffers which splits into some “mini batches”?

[Funatiq]

Hey Alex, the buffer management should be more clear in recent versions. However, I would recommend looking at the PyTorch backend which is the default since v1.0.

Runtime buffers are using for storing values for TRT engine inputs and outputs

Correct.

Decoder buffers, if I get right decoder means not actual decoder, rather than sampler that sample discrete tokens from models logins and doing all beam search logic which is needed

Correct. These have been refactored into DecoderState. There is only one of this.

Decoder inputs and outputs - still does not understand why do we need them, I we already have decoder buffers

The idea here is that inputs and outputs are only valid for a specific iteration. There can be multiple batches each with their own inputs and outputs.

Encoder buffers - why do we need them as well if encoder is running in separate model instance which has their own runtime buffers

Not sure. I think they are needed to store additional information that is not present in decoder-only models.

req slots - what is that?

req slot / batch slot is an identifier which maps a request to a specific resource slot. The slot is persistent for the whole execution of the request.

why executor instance have multiple buffers which splits into some “mini batches”?

Multiple batches (each with their own buffers) are used in TrtOverlap (overlap scheduling) and pipeline parallelism.

[alxmamaev]

@Funatiq Hi, just noticed your reply. What do you mean in PyTorch backend? I remember there was two options: Python Runtime and C++ runtime, and the reason why I was looking into C++ runtime was Continuous Batching support and ability to use C++ runtime via Triton Server.

[Funatiq]

Hi Alex, you are correct, this was the case some time ago, but we are now fully committed to the PyTorch backend (source files here) which is basically a complete rework and not related to the old Python runtime . Please note that the documentation has also been updated to reflect this.