Adding IT suite for PPL-based dashboards in Neo for CloudWatch Lake

[aalva500-prog]

Description

This PR adds integration tests and documentation for PPL based dashboards covering NFW, CloudTrail, WAF, and VPC logs in Neo for CloudWatch Lake.

Changes

New Integration Tests:

• Added NFW (Network Firewall) PPL dashboard integration tests
• Added CloudTrail PPL dashboard integration tests
• Added VPC Flow Logs PPL dashboard integration tests
• Added WAF PPL dashboard integration tests
• Reorganized PPL dashboard tests into dedicated dashboard/ package

Test Infrastructure:

• Added new index mappings and test data for NFW, CloudTrail, VPC, and WAF logs

Documentation:

• Added documentation for NFW, CloudTrail, VPC, and WAF PPL based dashboards integration tests

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[Swiddis]

question: Where exactly is this data from?

We had the issue with integrations that we committed a bunch of sample data and lost the source it was generated from. That would be a good thing to put in the docs.

[RyanL1997]

+1 We need to get some data from the actual real flow, like from the actual loggroup of VPC logs

[RyanL1997]

Let me actually follow up on this. I do have some existing flow that I can export some sanitized data. Cuz even for the verification of query correctness, we may need some field with relatively high cardinality.

[aalva500-prog]

The data you see here is coming from my own S3 bucket, which I use during the integration creation workflow in OpenSearch Dashboards. However, this is not real data and it only has a few records, as the purpose of this exercise was to test query correctness specifically. I had to change the data to avoid exposing sensitive info also.

[aalva500-prog]

During the new workflow in PPL based dashboards the data should come directly from CW log groups, though. Apart from the NFW, the rest of the queries have been already tested by DQS team connecting directly with CW log groups. However, for NFW the data has been retrieved directly from own CW log groups, so the schema is correct, but I can add some more data if needed.

[aalva500-prog]

That would be good, thanks @RyanL1997

Let me actually follow up on this. I do have some existing flow that I can export some sanitized data. Cuz even for the verification of query correctness, we may need some field with relatively high cardinality.