Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[eem] stats by metadata #204137

Closed
wants to merge 3 commits into from

Conversation

klacabane
Copy link
Contributor

@klacabane klacabane commented Dec 12, 2024

Metadata fields are currently aggregated with TOP which does not dedup values. This change introduces an initial STATS BY in the query that will group entities by their identity_fields + metadata_fields, creating unique set of values. A second (mostly unchanged) STATS BY will aggregate the metadata fields with a TOP(10) against this intermediate representation of the entities instead of the data source to retrieve (not guaranteed) more relevant data

new query collecting host.name per service.name

FROM logs-* |
STATS custom_timestamp_field = MAX(custom_timestamp_field) BY service.name::keyword, host.name::keyword |
RENAME `service.name::keyword` AS service.name, `host.name::keyword` AS host.name |
STATS host.name = TOP(host.name, 10, "ASC"), entity.last_seen_timestamp = MAX(custom_timestamp_field) BY service.name |
EVAL entity.type = "service", entity.id = service.name, entity.display_name = entity.id |
SORT entity.id DESC |
LIMIT 5

current query collecting host.name per service.name

FROM logs-* |
STATS host.name = TOP(host.name::keyword, 10, "ASC"), entity.last_seen_timestamp = MAX(custom_timestamp_field) BY service.name::keyword |
RENAME `service.name::keyword` AS service.name |
EVAL entity.type = "service", entity.id = service.name, entity.display_name = entity.id |
SORT entity.id DESC |
LIMIT 5

@klacabane klacabane self-assigned this Dec 12, 2024
@elasticmachine
Copy link
Contributor

elasticmachine commented Dec 12, 2024

🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

  • Click to trigger kibana-pull-request for this PR!
  • Click to trigger kibana-deploy-project-from-pr for this PR!

@klacabane klacabane added release_note:skip Skip the PR/issue when compiling release notes Team:obs-entities Observability Entities Team labels Dec 12, 2024
@elasticmachine
Copy link
Contributor

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

✅ unchanged

History

cc @klacabane

@klacabane klacabane closed this Dec 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release_note:skip Skip the PR/issue when compiling release notes Team:obs-entities Observability Entities Team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants