separator not working for pretty json

[llermaly]

Problem

I'm trying to parse pretty print JSON logs. Each file will contain an exact json object.

It works if I set the flush interval option, but I wonder why separator "" doesnt work.

Example log:

{
       "foo":"bar"
}

<source>
  @type tail
  tag vte.errorlogffa
  path /tmp/tm-logs/*
  read_from_head true
  <parse>
    @type none
  </parse>
</source>

<filter vte.errorlogffa>
  @type concat
  key message
  multiline_start_regexp /^{/
  multiline_end_regexp /^}/
  #flush_interval 5s
  separator ""
</filter>

<filter vte.errorlogffa>
  @type parser
  format json
  key_name message
</filter>

<match vte.errorlogffa>
  @type elasticsearch
  host elasticsearch
  port 9200
  index_name tm-errorlog-000001
  reconnect_on_error true
  <buffer>
    flush_thread_count 2
    flush_interval 1s
  </buffer>
</match>

This code throws:

2023-09-21 22:20:34 2023-09-22 03:20:34 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data '{  \"date\" ...

What I'm doing wrong?

Thanks
...

Steps to replicate

Provide example config and message

Expected Behavior

...

Your environment

• OS version
• paste result of fluentd --version or td-agent --version
• plugin version
  • paste boot log of fluentd or td-agent
  • paste result of fluent-gem list, td-agent-gem list or your Gemfile.lock

[daipom]

I think there is nothing wrong with fluent-plugin-concat.
It would be because the concatenated data is not the correct JSON format.

This code throws:

2023-09-21 22:20:34 2023-09-22 03:20:34 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data '{  \"date\" ...

Did you omit this log?
You can confirm the format of the data in this log.
Please check whether the data is correct JSON.

[llermaly]

Hi @daipom The error is when I add the "separator" parameter.

If I remove it and manually add a line break to the logfile it works but the original file doesnt contain a line break.

The content of the file is a json object in pretty format.

I would expect setting separator as "" will stop and create the event when it finds the closing }

Couldnt find any guidance on parsing pretty printed json objects