fluentd memory usage in buffer section

[rave189]

Describe the bug

We're collecting pod logs via fluentd in a Kubernetes environment.
We're using buffer sections to enhance the stability of log collection.
Checking the memory status of fluentd using the Grafana dashboard, we see that memory usage continues to increase as fluentd's runtime increases.
To resolve this situation, I upgraded the fluentd version to 1.18, but the issue persisted.
I ran several tests, but the only one that didn't cause a memory leak was the one that didn't use the buffer section.

To Reproduce

I modified some of the file-related code in buffer.rb because it seemed that closing a file after opening it was not working properly.
But nothing was solved.

Expected behavior

resolve memory leak

Your Environment

- Fluentd version: fluentd-kubernetes-daemonset:v1.14.6

Your Configuration

<buffer>
   @type file
   path <path>
   flush_at_shutdown true
   flush_mode interval
   flush_thread_count 16
   flush_interval 1
   chunk_limit_size 10m
   total_limit_size 65g
   queued_chunks_limit_size 4096
</buffer>

Your Error Log

not shown

Additional context

It is deployed as a daemonset on the k8s node, and the amount of logs collected per day exceeds 50GB.
I have an environment with less than 50gb, but the phenomenon was the same.

[daipom]

Thanks for your report!
It seems we need more info to investigate this issue.

If the flash is not keeping up and buffer files continue to accumulate, it may appear as though there is a memory leak.

It would be helpful to have a simple set of steps to reproduce the issue.

I ran several tests, but the only one that didn't cause a memory leak was the one that didn't use the buffer section.

Could you share your config?
How it behaves when the buffer section is not used depends on the plugin.

[Watson1978]

hmm, I can't reproduce the issue.
Can you attach your full configuration and provide the more info?

[rave189]

Here is the full fluentd config file.

<source>
    @type tail
    @label @log_output
    path <path>
    pos_file <pos file path>
    tag *
    read_from_head true
    <parse>
        @type none
    </parse>
</source>
 
<label @log_output>
    <match>
        <buffer>
            @type file
            path <path>
            flush_at_shutdown true
            flush_mode interval
            flush_thread_count 16
            flush_interval 1
            chunk_limit_size 10m
            total_limit_size 65g
            queued_chunks_limit_size 4096
        </buffer>
        @type forward
        <server>
            name <forward server name>
            host <ip address>
            port <port>
        </server>
    </match>
</label>

Additionally, here are some things I tested:

1. Add file close code (file_chunk.rb, buf_file.rb etc...)
2. upgrade image to v1.19.0-debian-opensearch-1.1
3. add path_suffix in buffer section
4. change buffer path
5. fix flush_interval 1 → 5m

I ran the above tests, but memory leaks occurred in all of them except for those that did not use a buffer.

[Watson1978]

Thanks.
I looked, but this is not a typical memory leak such as #4864.
The memory space used by Ruby objects was approximately 100MB.

In Ruby, the allocated memory area for used objects may be reused instead of being freed.
When fragmentation occurs in reused memory areas, Ruby might expand the necessary memory area.

A long-known workaround is to adjust MALLOC_ARENA_MAX (Ref. https://www.speedshop.co/2017/12/04/malloc-doubles-ruby-memory.html).
Or, you can use jemalloc.