Feature Request: View/Dump All Metadata Associated with OpenSearch Records

[neethu00]

Is your feature request related to a problem? Please describe.
In OpenSearch. Currently, users have to explicitly know or keep track of any metadata to use it when working with records in OpenSearch. This makes it difficult to access and utilize important information about how records were sourced, processed, and sinked to OpenSearch.

Describe the solution you'd like
Implement a feature that allows users to easily view or dump all metadata associated with a record in OpenSearch. This should include information about how the record was sourced, processed, and sinked to OpenSearch. The solution should provide a simple and intuitive way to access this metadata without requiring users to manually track or remember specific metadata fields.

Additional context

This feature would greatly improve the traceability and auditability of data in OpenSearch. It would be particularly useful for debugging, data lineage tracking, and ensuring compliance with data governance policies. The ability to easily access and view all metadata associated with a record would enhance the overall usability and value of the OpenSearch system for data management and analysis tasks.

[dlvenable]

@neethu00 , Thank you for opening this issue.

We are looking for some clarity on the ask here. Are you wanting this to help debug Data Prepper pipelines? Or this more to have a full audit history of events?

Do you want this metadata to be stored in OpenSearch? Or do you want them in the logs?

Can you elaborate on what metadata you are looking to include?

[neethu00]

@dlvenable
Please find the customer expection below:

The expectation is that I can output the logEvents field AND be able to identify the source SubscriptionFilter through the subscriptionFilters key to include in the record i'm outputting to OpenSearch. Is this possible