[eem] stats by metadata

[klacabane]

Metadata fields are currently aggregated with TOP which does not dedup values. This change introduces an initial STATS BY in the query that will group entities by their identity_fields + metadata_fields, creating unique set of values. A second (mostly unchanged) STATS BY will aggregate the metadata fields with a TOP(10) against this intermediate representation of the entities instead of the data source to retrieve (not guaranteed) more relevant data

new query collecting host.name per service.name

FROM logs-* |
STATS custom_timestamp_field = MAX(custom_timestamp_field) BY service.name::keyword, host.name::keyword |
RENAME `service.name::keyword` AS service.name, `host.name::keyword` AS host.name |
STATS host.name = TOP(host.name, 10, "ASC"), entity.last_seen_timestamp = MAX(custom_timestamp_field) BY service.name |
EVAL entity.type = "service", entity.id = service.name, entity.display_name = entity.id |
SORT entity.id DESC |
LIMIT 5

current query collecting host.name per service.name

FROM logs-* |
STATS host.name = TOP(host.name::keyword, 10, "ASC"), entity.last_seen_timestamp = MAX(custom_timestamp_field) BY service.name::keyword |
RENAME `service.name::keyword` AS service.name |
EVAL entity.type = "service", entity.id = service.name, entity.display_name = entity.id |
SORT entity.id DESC |
LIMIT 5

[elasticmachine]

🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

• Click to trigger kibana-pull-request for this PR!
• Click to trigger kibana-deploy-project-from-pr for this PR!

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 008c9ec

Failed CI Steps

• Jest Tests #1

Metrics [docs]

✅ unchanged

History

• 💔 Build #260322 failed 18d09b1

cc @klacabane