-
Notifications
You must be signed in to change notification settings - Fork 625
Description
Describe the Bug
The fix in #5638 (issue #5637) excludes required_fields from the rule hash when an ES|QL rule uses | keep *, so that stack-dependent required fields don’t cause artificial version bumps.
Currently we only treat the literal * as a wildcard. In ES|QL, any keep value that ends with * is a wildcard (e.g. cloud.*, event.*). Those patterns also lead to required fields that depend on integration/schema and stack version, so they are equally non-deterministic for hashing. We should exclude required_fields from the hash whenever any keep field is a wildcard—i.e. either exactly * or any field ending with *.
To Reproduce
- Use or add an ES|QL rule whose query has
| keepwith a field that ends in*(e.g.| keep cloud.*, event.agent_id). - Run the lock-version workflow (or equivalent hash calculation).
- Observe that
required_fieldscan still change with stack/validation and cause unnecessary version bumps, because the current check only looks for the literal*in the keep list.
Expected Behavior
- For ES|QL rules,
required_fieldsshould be excluded from the hash whenever the| keepclause contains any wildcard: either the literal*or any field whose name ends with*. - Rules that use only non-wildcard keep fields (e.g.
| keep event.agent_id, host.name) should continue to includerequired_fieldsin the hash.
Implementation (diff vs main)
Change in detection_rules/rule.py: extend _uses_keep_star() so it treats both “keep ” and “keep …” as wildcards.
Diff of detection_rules/rule.py vs main:
--- a/detection_rules/rule.py
+++ b/detection_rules/rule.py
@@ -1261,7 +1261,7 @@ class BaseRuleContents(ABC):
return obj
def _uses_keep_star(self, hashable_dict: dict[str, Any]) -> bool:
- """Check if this is an ES|QL rule that uses `| keep *`."""
+ """Check if this is an ES|QL rule that uses `| keep *` or fields ending with '*'."""
if hashable_dict.get("language") != "esql":
return False
@@ -1273,7 +1273,7 @@ class BaseRuleContents(ABC):
keep_match: re.Match[str] | None = keep_pattern.search(query)
if keep_match:
keep_fields: list[str] = [field.strip() for field in keep_match.group(1).split(",")]
- return "*" in keep_fields
+ return any(field == "*" or field.endswith("*") for field in keep_fields)
return False- Docstring: Updated to state we also treat fields ending with
*as wildcards. - Logic: Replaced
return "*" in keep_fields(which only matched when a keep field is exactly"*") withreturn any(field == "*" or field.endswith("*") for field in keep_fields), so we also match patterns likecloud.*orevent.*, ensuringrequired_fieldsare excluded from the hash for all wildcard keep usage.
Screenshots
DIff between 9.2 and 9.3 no longer causes a bump due to ECS fields where as before it did:
Desktop - OS
None
Desktop - Version
No response
Additional Context
- Same motivation as #5637:
required_fieldsderived from wildcard keep patterns depend on integration schemas and stack version, so they should not drive the rule hash. - No change to
get_hashable_content()or call sites; only the condition inside_uses_keep_star()is broadened.