Data streams and aliases manipulation

[barebu]

Hi. Try to delete data streams with curator 8.0.21, get logs with 404 like:

2025-06-02 16:09:20,333 INFO      HEAD http://127.0.0.1:9200/_alias/.ds-prod-2025.04.21-2025.05.21-000002 [status:404 duration:0.005s]
2025-06-02 16:09:20,338 INFO      HEAD http://127.0.0.1:9200/_alias/.ds-staging-2025.04.30-2025.05.30-000002 [status:404 duration:0.004s]
2025-06-02 16:09:20,344 INFO      HEAD http://127.0.0.1:9200/_alias/.ds-prod-2025.05.10-2025.05.10-000001 [status:404 duration:0.005s]

config:

---
elasticsearch:
  client:
    hosts: http://127.0.0.1:9200
    verify_certs: false
    client_cert:
    client_key:
logging:
  loglevel: INFO
  logfile: "/var/log/curator"
  logformat: default
  blacklist:
  - elasticsearch
  - urllib3

actions:

   action: delete_indices
    description: Delete all indices older N days by creation date
    options:
      disable_action: false
      ignore_empty_list: true
      continue_if_exception: true
      include_hidden: true
    filters:
    - filtertype: pattern
      kind: regex
      value: "^(?!\\.).*|(.ds-).*$"
    - filtertype: age
      source: creation_date
      direction: older
      timestring: "%Y.%m.%d"
      unit: days
      unit_count: '30'
    - filtertype: kibana
      exclude: true

what am I doing wrong)

[untergeek]

Curator is trying to assert that the index name it matched is not an alias. The test is "failing" (404) because the index name is not an alias. It's logged as INFO. This is not an error, but confirming that the index is not an alias.

This is not causing problems for the delete. If you are having an issue with the delete, it is from something else, not this.

[barebu]

@untergeek Thanks for such a quick reply. Can curator clean up data streams in general? It seems that the reason indexes are not deleted is because they are tied to streams

[untergeek]

Indices from data_streams can be deleted by Curator. It will not delete an index that is still the current "write" index of a data_stream, however, regardless of whether the index meets the name/age/pattern requirements.

Out of curiosity, why are you using Curator for data_streams instead of using ILM (Index Lifecycle Management)?

[barebu]

Curator is still a more powerful and flexible tool for rotating elasticsearch entities than ILM in our opinion. In addition, curator's configuration is better and more conveniently suited to our infrastructure as code approach.
Are there any plans to add data streams management via curator in the future?

[untergeek]

Short answer: Yes.
Long answer: There are things in motion that will hopefully be done by the end of the summer. Wait and see.