Add max_lines parameter to limit total message size

[dshackith]

Suggestion

Add max_lines parameter used along with partial_key to limit the number of lines concatenated into a single message. Possible max_behavior parameter to indicate what to do when max is hit (truncate, new, drop).

Rationale

Sometimes messages that are broken up by Docker and reassembled by Fluentd via concat are too big for a later part of the pipeline. The reason why Docker breaks up messages is to avoid OOME. If we reconstruct the full message, it is possible for this to trigger an OOME in Fluentd, especially if there are several or many. Using max_lines to limit the total number of lines concatenated effectively acts as a truncation device on logs that exceed a certain length. In my case, messages from Docker are broken into 16k, but I emit to Kafka which we limit to 1M. With additional metadata/headers added, we need to limit the maximum number of lines to ~57 to avoid overrunning the limit on Kafka.

Example config

<filter>
      @type concat
      stream_identity_key docker_id
      key message
      partial_key partial_message
      partial_value true
      separator ""
      flush_interval 5
      max_lines 57
      max_behavior truncate
 </filter>

[okkez]

Could you try #68 ?

[dshackith]

Unfortunately I misunderstood how and where the partial metadata was implemented in the Docker logger. Currently we are using fluent-bit to forward logs to fluentd, so we are using the json-file logger in Docker which does not include any partial metadata yet (see moby/moby#36777). Because I am not using the fluentd logging driver in Docker which does include this metadata, I am unable to test this with partial metadata.
That said, this functionality would be useful for the regexp options as well.
I apologize for not fleshing this out more before opening the issue.

[okkez]

OK, thank you for describing your background and current status.
If someone wants to use this new functionality, we can merge the PR.

[dshackith]

This functionality applied to regexp would still be useful for my use case.

[yaroslav-shcherbyniuk]

@okkez We would love to have this functionality to regexp instead of partial_key

<filter var.log.**>
  @id filter_concat
  @type concat
  key log
  use_first_timestamp true
  multiline_end_regexp /\n$/
  separator ""
  buffer_limit_size 200000
  buffer_overflow_method new
 </filter>

[Watson1978]

Please try v2.6.0, it contains #68 feature.

Thanks.