Skip to content

[Bug] ES|QL hash calc should ignore all keep wildcards (not only | keep *) #5764

@eric-forte-elastic

Description

@eric-forte-elastic

Describe the Bug

The fix in #5638 (issue #5637) excludes required_fields from the rule hash when an ES|QL rule uses | keep *, so that stack-dependent required fields don’t cause artificial version bumps.

Currently we only treat the literal * as a wildcard. In ES|QL, any keep value that ends with * is a wildcard (e.g. cloud.*, event.*). Those patterns also lead to required fields that depend on integration/schema and stack version, so they are equally non-deterministic for hashing. We should exclude required_fields from the hash whenever any keep field is a wildcard—i.e. either exactly * or any field ending with *.

To Reproduce

  1. Use or add an ES|QL rule whose query has | keep with a field that ends in * (e.g. | keep cloud.*, event.agent_id).
  2. Run the lock-version workflow (or equivalent hash calculation).
  3. Observe that required_fields can still change with stack/validation and cause unnecessary version bumps, because the current check only looks for the literal * in the keep list.

Expected Behavior

  • For ES|QL rules, required_fields should be excluded from the hash whenever the | keep clause contains any wildcard: either the literal * or any field whose name ends with *.
  • Rules that use only non-wildcard keep fields (e.g. | keep event.agent_id, host.name) should continue to include required_fields in the hash.

Implementation (diff vs main)

5764_bugfix.patch

Change in detection_rules/rule.py: extend _uses_keep_star() so it treats both “keep ” and “keep …” as wildcards.

Diff of detection_rules/rule.py vs main:

--- a/detection_rules/rule.py
+++ b/detection_rules/rule.py
@@ -1261,7 +1261,7 @@ class BaseRuleContents(ABC):
         return obj
 
     def _uses_keep_star(self, hashable_dict: dict[str, Any]) -> bool:
-        """Check if this is an ES|QL rule that uses `| keep *`."""
+        """Check if this is an ES|QL rule that uses `| keep *` or fields ending with '*'."""
         if hashable_dict.get("language") != "esql":
             return False
 
@@ -1273,7 +1273,7 @@ class BaseRuleContents(ABC):
         keep_match: re.Match[str] | None = keep_pattern.search(query)
         if keep_match:
             keep_fields: list[str] = [field.strip() for field in keep_match.group(1).split(",")]
-            return "*" in keep_fields
+            return any(field == "*" or field.endswith("*") for field in keep_fields)
         return False
  • Docstring: Updated to state we also treat fields ending with * as wildcards.
  • Logic: Replaced return "*" in keep_fields (which only matched when a keep field is exactly "*") with return any(field == "*" or field.endswith("*") for field in keep_fields), so we also match patterns like cloud.* or event.*, ensuring required_fields are excluded from the hash for all wildcard keep usage.

Screenshots

DIff between 9.2 and 9.3 no longer causes a bump due to ECS fields where as before it did:

Image

Desktop - OS

None

Desktop - Version

No response

Additional Context

  • Same motivation as #5637: required_fields derived from wildcard keep patterns depend on integration schemas and stack version, so they should not drive the rule hash.
  • No change to get_hashable_content() or call sites; only the condition inside _uses_keep_star() is broadened.

Metadata

Metadata

Labels

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions