Describe the bug
When a batch can not fit a message, kafka waits until the current batch is delivered to add the message to the next batch. Linger time prevents batches from being delivered until a cooldown period has expired. Combined this makes it so that large messages time out.
Expected behaviour
From what I understand about linger time, its meant to set a maximum wait time until the message is to be delivered to message broker. Its not meant to act as a rate limit, but instead a mechanism to ensure a few large packets are sent out rather than many small ones.
Environment (please complete the following information):
- aiokafka version (
python -c "import aiokafka; print(aiokafka.__version__)"): 0.12.0
- Kafka Broker version (
kafka-topics.sh --version): 4.0.0
Reproducible example
I used the files in this gist for looking into the problem. The python program was for testing sending many large messages, the json file was the data we were sending, and the patch was something I created to look at how aiokafka was sending it.
Describe the bug
When a batch can not fit a message, kafka waits until the current batch is delivered to add the message to the next batch. Linger time prevents batches from being delivered until a cooldown period has expired. Combined this makes it so that large messages time out.
Expected behaviour
From what I understand about linger time, its meant to set a maximum wait time until the message is to be delivered to message broker. Its not meant to act as a rate limit, but instead a mechanism to ensure a few large packets are sent out rather than many small ones.
Environment (please complete the following information):
python -c "import aiokafka; print(aiokafka.__version__)"): 0.12.0kafka-topics.sh --version): 4.0.0Reproducible example
I used the files in this gist for looking into the problem. The python program was for testing sending many large messages, the json file was the data we were sending, and the patch was something I created to look at how
aiokafkawas sending it.