out_forward doesn't flush file buffers

[TheDaz-1]

Describe the bug

After a failure the system was restarted but fluentd never started working the disk buffers. Only after I changed the flush configuration to flush_mode immediate it immediately started to flush the file buffers.

To Reproduce

can't reproduce at will

Expected behavior

fluentd should automatically flush file buffers after a restart

Your Environment

OS     : RHEL 9
Kernel : 5.14.0-427.42.1
fluentd: 1.16.1

Your Configuration

@type forward
     <server>
        name myServer
        host 127.0.0.1
        port 24225
        ignore_network_errors_at_startup true
        verify_connection_at_startup false
     </server>
     <buffer>
        @type file
        path /ffdc/fluent/buffers/mybuf.*.buf
        flush_mode interval
        flush_interval 1s
        overflow_action block
        total_limit_size 1GB
        queued_chunks_limit_size 1024
        compress gzip
        retry_forever true
        retry_type periodic
        retry_wait 1s
     </buffer>

Your Error Log

no error logs written

Additional context

No response

[kenhys]

FYI: if you want flush manually you can use SIGUSR1

ref. https://docs.fluentd.org/deployment/signals#sigusr1

[Watson1978]

Please let us know if you find out the reproduction conditions.

[TheDaz-1]

FYI: if you want flush manually you can use SIGUSR1

ref. https://docs.fluentd.org/deployment/signals#sigusr1

thx, will try next time

[TheDaz-1]

Please let us know if you find out the reproduction conditions.

what data should I collect in case I come across this again ?

[Watson1978]

Can your environment displays failed to write data into buffer by buffer overflow warning message?
Or, can your environment displays the error log if you change overflow_action block to overflow_action throw_exception ?

Or, you need to change buffer time_key, such as

  <buffer []> 
    ...
  </buffer>

[TheDaz-1]

Can your environment displays failed to write data into buffer by buffer overflow warning message? Or, can your environment displays the error log if you change overflow_action block to overflow_action throw_exception ?

fluentd is running as a systemd service. Would it write to the /var/log/messages file in case of a failure ? systemd starts the service with the --no-supervisor --log <log_file_name> option. Would it write to the log file specified ? I'll change the config to let fluentd throw an exception in case of a buffer overflow.

[Watson1978]

I think it would be output to the same location as the other logs handled by Fluentd.

[daipom]

systemd starts the service with the --no-supervisor --log <log_file_name> option. Would it write to the log file specified ?

Yes.

[TheDaz-1]

So, it hit again. This time the fluentd out_forward plugin started sending invalid packets consisting of the hex sequence

93 a5 7a 2e 6c 6f 67

and then closed the connection. I did so over and over again while at the same time the disk buffer started piling up logs. The fluentd log file was empty and only showed :

# Logfile created on 2025-03-01 09:38:18 -0500 by logger.rb/v1.4.3

[TheDaz-1]

Is there anyone who could explain why fluentd keeps sending these invalid requests and how to fix/avoid it ? Once fluentd enters this state there's no escape other than stopping fluentd altogether and deleting all file buffers thereby losing important information.

[TheDaz-1]

Happened again. fluentd stopped flushing the buffers. Not even a reboot solved the issue. The only thing that helps is explicitly sending a SIGUSR1 signal to the fluentd process. The server log was empty.