What is the problem you are trying to solve?
When writing to ingest storage, the distributor retains decoded request buffers and timeseries while waiting for Kafka acknowledgements. Once serialization is complete, the Kafka client only needs the encoded records.
Which solution do you envision (roughly)?
Release the decoded data once all consumers have finished using it. This should reduce distributor heap usage, particularly when backend writes are slow.
Keep request completion and inflight accounting separate from buffer cleanup, and preserve the existing lifetime for paths that still need decoded data, such as dual writes to ingesters.
Validate correctness and compare memory usage with delayed Kafka acknowledgements.
Locally tested before/after of proposed fix #16676 with a five-second Kafka broker stall: measured 72–89% lower median peak allocated heap and 66–84% lower process RSS across three workloads
What is the problem you are trying to solve?
When writing to ingest storage, the distributor retains decoded request buffers and timeseries while waiting for Kafka acknowledgements. Once serialization is complete, the Kafka client only needs the encoded records.
Which solution do you envision (roughly)?
Release the decoded data once all consumers have finished using it. This should reduce distributor heap usage, particularly when backend writes are slow.
Keep request completion and inflight accounting separate from buffer cleanup, and preserve the existing lifetime for paths that still need decoded data, such as dual writes to ingesters.
Validate correctness and compare memory usage with delayed Kafka acknowledgements.
Locally tested before/after of proposed fix #16676 with a five-second Kafka broker stall: measured 72–89% lower median peak allocated heap and 66–84% lower process RSS across three workloads