Flatten json processor

[oeyh]

Is your feature request related to a problem? Please describe.
There're use cases where nested json objects needs to be flattened. For example:
From:

{
  "key1": "val1",
  "key2": {
    "key3": {
      "key4": "val2"
    }
  },
  "list1": [
    {
      "list2": [
        {
          "name": "name1",
          "value": "value1"
        },
        {
          "name": "name2",
          "value": "value2"
        }
      ]
    }
  ]
}

to

{
  "key1": "val1",
  "key2.key3.key4": "val2",
  "list1[0].list2[0].name": "name1",
  "list1[0].list2[0].value": "value1",
  "list1[0].list2[1].name": "name2",
  "list1[0].list2[1].value": "value2",
}

In some specific use cases (#3965), the users wants to remove list indices from the flattened keys and combine leaf values in lists:

{
  "key1": "val1",
  "key2.key3.key4": "val2",
  "list1[].list2[].name": ["name1","name2"],
  "list1[].list2[].value": ["value1","value2"]
}

Describe the solution you'd like
A new flatten processor with these configurations:

  processor:
    - flatten:
        source: ""   
        target: ""  
        remove_processed_fields: true
        remove_list_indices: false

• source: the source key to the object to flatten
• target: the target key to put the flatten object
• remove_processed_fields: boolean, whether to remove processed fields, only keeping flattened fields
• remove_list_indices: boolean, whether to remove list indices from the flattened keys and combine leaf values

Describe alternatives you've considered (Optional)
N/A

Additional context
A specific use case is in #3965

[kkondaka]

@oeyh This does not mean we should take the suggestion literally. I do not think we should have "[]" in keys. I serves no purpose.

[timo-mue]

Sorry if this is the wrong place to ask, since this issue is already closed and the processor implemented, but what exactly is the purpose of having the empty squared brackets as part of the key when remove_list_indices is true?
We use the flatten processor with this option and for our usecase it would be much better and I think in general more intuitive if we had "list1.list2.name": ["name1","name2"] instead of "list1[].list2[].name": ["name1","name2"]

[oeyh]

@timo-mue It was based on a use case in #3965. Feel free to open an issue if you have different use cases and requests.

[timo-mue]

@timo-mue It was based on a use case in #3965. Feel free to open an issue if you have different use cases and requests.

thanks for your answer. I opened a new issue