Batching and Buffering - inter dependencies

[hemanthofficial3009]

hello team,
We have implemented the buffering and batching concepts on the vector sink instance.
our sample configuration:
[sinks.forward]
type = "vector"
inputs = ["validator"]
address = "ABC:1234"
buffer.type = "disk"
buffer.max_size = 50000
version = "2"
batch.max_bytes = 50000
batch.timeout_secs = 60

in this case we have tried to test by exceeding the batch.max_bytes before the batch.timeout_secs. we expected the sink to flush the data to the downstream vector source instance which did not happen.

can you please elaborate on these parameters for buffering and batching

1. how are related internally?
2. how do they behave when the max limits are met each of them?
3. data type of buffer.max_size is it in bytes ?
4. When does the vector sink instance flush the data to downstream ?

[jszwedko]

Hi @hemanthofficial3009 !

can you please elaborate on these parameters for buffering and batching

1. how are related internally?

Buffering happens before batching. For disk buffers, the data will remain in the buffer until it has been processed by the sink and sent to the target.

2. how do they behave when the max limits are met each of them?

For disk buffers, it will block by default meaning that it will propagate back-pressure up to any connected components. This behavior can be changed via configuring when_full. The only other option currently is drop_newest which will drop new events when the buffer is full.

For batches, when the batch is full it will be flushed downstream.

3. data type of buffer.max_size is it in bytes ?

That's correct, it is bytes. We should probably add _bytes to the name like we do for batch.

4. When does the vector sink instance flush the data to downstream ?

When the batch is either full (exceeds max_bytes) or is older than timeout_secs (that is: the first event in the batch was received more than timeout_secs seconds ago) it will be flushed.

You might be seeing some odd behavior by setting the disk buffer max size to the same as the max batch size as the max batch size is actually based on the in-memory size of the events (ref: #10020) whereas the disk buffer max size is based on the number of serialized bytes for the events (typically this is lower than the in-memory representation). Even given that, the behavior you are describing does sound incorrect though. It shouldn't be the case that the disk buffer would fill up and block before the batch was full. If you are able to reproduce this, could you open a bug report with configuration and reproduction steps?