feat: add ML calendar and calendar event resources - #1969
Conversation
|
✅ PR Changelog Check passed — the |
Add two new Terraform resources for managing Elasticsearch ML calendars: - `elasticstack_elasticsearch_ml_calendar` — manages calendar lifecycle and job associations via individual PutCalendarJob/DeleteCalendarJob endpoints for in-place job_ids updates. - `elasticstack_elasticsearch_ml_calendar_event` — manages individual scheduled events with RFC3339 time handling and server-generated event IDs. All attributes require replacement (no update API). Both resources support import and follow the existing Plugin Framework patterns. Includes requirements doc, unit tests (16 cases), acceptance tests (5 cases), and generated documentation. Made-with: Cursor
Made-with: Cursor
Replace defer with immediate Close() calls inside the job add/remove loops to avoid keeping all response bodies open until function return. Made-with: Cursor
- Break long lines in schema descriptions to stay under 200 char limit (lll) - Rename exported types to avoid stuttering (revive): CalendarTFModel→TFModel, CalendarCreateAPIModel→CreateAPIModel, CalendarAPIModel→APIModel - Fix gofmt alignment after renames Made-with: Cursor
- Use ElasticsearchResource envelope, scoped client, and Ml typed APIs for calendar and calendar_event. - Import state sets resource id attributes; composite IDs support nested calendar_id/event_id segments. - Regenerate resource docs; note new resources in CHANGELOG. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Pull request overview
Adds two new Terraform Plugin Framework resources to manage Elasticsearch ML anomaly detection calendars and their scheduled events, including provider registration, docs, and test coverage.
Changes:
- Register new ML calendar and calendar event resources in the provider.
- Implement
elasticstack_elasticsearch_ml_calendarwith CRUD + in-place job association reconciliation on update. - Implement
elasticstack_elasticsearch_ml_calendar_eventwith create/read/delete + composite IDs and RFC3339 time handling, plus unit/acceptance tests and generated docs.
Reviewed changes
Copilot reviewed 30 out of 30 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| provider/plugin_framework.go | Registers the new ML calendar and calendar event resources. |
| internal/entitycore/resource_envelope.go | Adjusts Elasticsearch envelope ID parsing to support resource IDs that may contain slashes. |
| internal/clients/api_client.go | Adds composite-ID parsing variant that splits only on the first / for Elasticsearch resources. |
| internal/elasticsearch/ml/calendar/resource.go | Introduces the ML calendar resource wrapper and import handling. |
| internal/elasticsearch/ml/calendar/schema.go | Defines calendar schema (calendar_id, description, job_ids) and validation. |
| internal/elasticsearch/ml/calendar/models.go | Adds TF/API models and mapping logic (including null/empty handling). |
| internal/elasticsearch/ml/calendar/create.go | Implements calendar create via PUT calendar API and sets composite id. |
| internal/elasticsearch/ml/calendar/read.go | Implements calendar read via GetCalendars and drift removal behavior. |
| internal/elasticsearch/ml/calendar/update.go | Implements update to reconcile job_ids associations via add/remove job endpoints. |
| internal/elasticsearch/ml/calendar/delete.go | Implements calendar delete via DeleteCalendar with 404 idempotency. |
| internal/elasticsearch/ml/calendar/models_test.go | Unit tests for calendar model conversions and null/empty semantics. |
| internal/elasticsearch/ml/calendar/acc_test.go | Acceptance tests for calendar CRUD/update, no-jobs behavior, and import. |
| internal/elasticsearch/ml/calendar/testdata/TestAccResourceMLCalendar/create/calendar.tf | Acceptance test config for calendar create case. |
| internal/elasticsearch/ml/calendar/testdata/TestAccResourceMLCalendar/update/calendar.tf | Acceptance test config for calendar update (job association) case. |
| internal/elasticsearch/ml/calendar/testdata/TestAccResourceMLCalendarNoJobs/create/calendar.tf | Acceptance test config for calendar with no jobs set. |
| internal/elasticsearch/ml/calendar/testdata/TestAccResourceMLCalendarImport/create/calendar.tf | Acceptance test config for calendar import scenario. |
| internal/elasticsearch/ml/calendar_event/resource.go | Introduces calendar event resource with custom Create/Update and import handling. |
| internal/elasticsearch/ml/calendar_event/schema.go | Defines calendar event schema with RFC3339 time custom types and replace semantics. |
| internal/elasticsearch/ml/calendar_event/models.go | Adds TF/API models and time conversions (RFC3339 ↔ epoch millis). |
| internal/elasticsearch/ml/calendar_event/create.go | Implements event creation via PostCalendarEvents and discovers server-generated event_id. |
| internal/elasticsearch/ml/calendar_event/read.go | Implements event read by listing events and locating the matching event_id. |
| internal/elasticsearch/ml/calendar_event/delete.go | Implements event delete via DeleteCalendarEvent with 404 idempotency. |
| internal/elasticsearch/ml/calendar_event/models_test.go | Unit tests for time conversions and composite ID parsing helpers. |
| internal/elasticsearch/ml/calendar_event/acc_test.go | Acceptance tests for calendar event create and import flows. |
| internal/elasticsearch/ml/calendar_event/testdata/TestAccResourceMLCalendarEvent/create/calendar_event.tf | Acceptance test config for event creation. |
| internal/elasticsearch/ml/calendar_event/testdata/TestAccResourceMLCalendarEventImport/create/calendar_event.tf | Acceptance test config for event import scenario. |
| docs/resources/elasticsearch_ml_calendar.md | Generated docs for the ML calendar resource schema. |
| docs/resources/elasticsearch_ml_calendar_event.md | Generated docs for the ML calendar event resource schema. |
| dev-docs/requirements/elasticsearch/ml/calendar.md | Adds implementation requirements/spec for both new resources. |
| CHANGELOG.md | Adds changelog entry for the two new ML resources (and normalizes formatting in the edited section). |
…ation) - Align calendar_id validators with ml_calendar (length + regex). - Add ValidateConfig so end_time must be strictly after start_time at plan time. - Regenerate resource documentation. Co-authored-by: Cursor <cursoragent@cursor.com>
PostCalendarEventsRequest/Response and GetCalendarEventsResponse were leftover from the low-level client; create/read use typed API structs. Co-authored-by: Cursor <cursoragent@cursor.com>
- Fail if listing events before POST fails (no silent empty snapshot). - Prefer event_id from PostCalendarEvents response when present. - On list diff, disambiguate multiple new events via description and times. - Add tests for time coercion and plan matching helpers. Co-authored-by: Cursor <cursoragent@cursor.com>
Drop the old dev-docs requirements artifact now that OpenSpec is the canonical requirements source. Co-authored-by: Cursor <cursoragent@cursor.com>
Resolve CHANGELOG [Unreleased] conflict: keep Kibana dashboard breaking change from main and ML calendar resources entry from this branch. Address Copilot review for ML calendar resources: - Paginate ML calendar event list/read/create paths (from/size) instead of a single 10k cap; treat first-page 404 as empty on read. - Parse event start/end times from multiple API representations (int64, json.Number, RFC3339 string, typed DateTime). - Preserve empty calendar description in state to avoid drift with description = "". - Extend unit tests accordingly. Co-authored-by: Cursor <cursoragent@cursor.com>
- ACC: invalid calendar_id (regex and length), bad import IDs, event time ordering, invalid event calendar_id, event import ID format. - Unit: readCalendar empty resource ID; split/parse composite ID edge cases; calendarEventAnyTimeToUnixMilli and fromAPIModel for json.Number, offsets, unsupported types; walkMLCalendarEventPagesWith pagination and errors. - Refactor event paging behind mlCalendarEventsPageFetcher for testability. Co-authored-by: Cursor <cursoragent@cursor.com>
…port ACC
- Set description to optional+computed with StaticString("") default so plan
matches refresh when the API returns an empty description (fixes inconsistent
result after apply for nested ml_calendar blocks).
- Import error ACC configs: embed elasticsearch.endpoints from ELASTICSEARCH_ENDPOINTS
when set so post-test destroy still has a configured client; fall back to the
empty elasticsearch block when the env var is absent (TF_ACC off / skipped).
Co-authored-by: Cursor <cursoragent@cursor.com>
…Persist terraform-plugin-testing replaces the main working dir config with a provider-only stub before import; with ImportStatePersist=false import runs in a temp dir, leaving the stub for post-test destroy and no elasticsearch block. Set ImportStatePersist on wrong-ID import steps and drop the env-based endpoints helper (no longer needed). Co-authored-by: Cursor <cursoragent@cursor.com>
…8.16 Schema defaults forced skip_result/skip_model_update into every create, which used the raw JSON path and caused 400 unknown field errors on older clusters (e.g. CI 8.3.x). Remove those defaults so unset attributes omit from the request; add a create-time version guard when optional scheduling fields are set, skip optional-fields ACC coverage below 8.16, and refresh generated docs. Co-authored-by: Cursor <cursoragent@cursor.com>
- CompositeIDFromStr: allow an empty cluster segment when the resource segment is non-empty so legacy Kibana synthetics IDs like "/<id>" still parse (fixes post-apply refresh on newer matrix shards). - Reject only a missing resource segment (e.g. trailing slash). - ML filter import-failure ACC: use cluster_uuid/ as the invalid id; two non-empty segments are valid with SplitN and no longer surface Wrong resource ID. - Extend synthetics TryReadCompositeID test for the leading-slash case. Co-authored-by: Cursor <cursoragent@cursor.com>
…cc test Plannable import (ImportBlockWithID) runs terraform state rm on step > 1; after Destroy the resource is absent from state, so the harness fails with Invalid target address. Use default ImportCommandWithID for the final nonexistent-filter import step only. Co-authored-by: Cursor <cursoragent@cursor.com>
…update callbacks Revert ElasticsearchUpdateFunc and KibanaUpdateFunc to a single planned model argument. Elasticsearch Update reuses writeFromPlan (read-after-write) like Create; Kibana Update no longer decodes req.State before the callback. Removes the wide mechanical prior-model parameter from envelope-backed resources and updates package doc. Drops the Kibana state.Get short-circuit test that only applied when prior state was decoded in Update. Co-authored-by: Cursor <cursoragent@cursor.com>
These write*Update helpers only forwarded to the create/write callback after the prior-state signature experiment; main already passes the same function for create and update. Restores script, logstash, slm, snapshot_repository, enrich, ingest pipeline, and security role/mapping/systemuser to upstream wiring so they disappear from the PR diff. Co-authored-by: Cursor <cursoragent@cursor.com>
…osite ID test deltas - Reset kibana_resource_envelope, maintenance_window, and streams to upstream/main so the PR does not carry Kibana envelope API churn. - Restore entitycore package doc Kibana paragraph to match main. - Adjust only kibana_resource_envelope_test (multi-slash read) and synthetics schema_test expectations for shared clients.CompositeIDFromStr first-slash parsing required by ML calendar composite IDs. Co-authored-by: Cursor <cursoragent@cursor.com>
…t wrapper) Same pattern as other envelope resources: main passes createDataStream for both callbacks; the updateDataStream helper only forwarded to create and was not needed for ML calendar work. Co-authored-by: Cursor <cursoragent@cursor.com>
…synthetics tests Comments tie assertions to clients.CompositeIDFromStr (first-slash split, legacy /<resource>) so drift from upstream test expectations is intentional and traceable. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…ope paths - Restore CompositeIDFromStr to historical two-segment Split semantics for Kibana, synthetics, and other global callers. - Implement nested parsing (first slash + legacy /<resource>) only in CompositeIDFromStrForElasticsearch, used from entitycore Elasticsearch Read/Delete via CompositeIDFromStrForElasticsearchFw. - Revert kibana envelope and synthetics schema tests to match main; restore ML filter import failure case id that is invalid under the global parser. Co-authored-by: Cursor <cursoragent@cursor.com>
- CHANGELOG: note envelope vs global composite ID parsing for reviewers. - entitycore: comment why create/update stubs split in envelope tests. - calendar_event: document placeholder create + Create override wiring. - acc tests: clarify SkipIfUnsupported vs other acc coverage for 8.16 fields. Co-authored-by: Cursor <cursoragent@cursor.com>
Resolve CHANGELOG: keep [Unreleased] ML calendar entries and insert [0.15.1] from upstream. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 49 out of 49 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (2)
internal/elasticsearch/ml/calendar_event/schema.go:1
- The regex alternation
|^[a-z0-9]$is redundant because the first branch already matches a single character due to the optional group. Simplifying to a single anchored expression (e.g.,^[a-z0-9]([a-z0-9_-]*[a-z0-9])?$) reduces maintenance and the chance of future divergence.
internal/elasticsearch/ml/calendar_event/wire_events.go:1 - With
json.Unmarshalintoany, numeric JSON values decode asfloat64(notjson.Number), so thecase json.Number:branch is effectively unreachable in normal usage. Either remove that case, or switch to ajson.DecoderwithUseNumber()so thejson.Numberhandling is exercised intentionally.
- Add ModifyPlan on ml_calendar and ml_calendar_event so a changed elasticsearch_connection block forces replacement (avoids silent wrong-cluster state with no-op update). - Defer listing calendar events during create until the POST response cannot supply event_id; resolve via a single paged scan for events matching the plan, with an error when multiple matches exist. - Simplify calendar_id regex to a single anchored pattern on both resources. - Drop unreachable json.Number branch in force_time_shift decode (default json.Unmarshal uses float64). Co-authored-by: Cursor <cursoragent@cursor.com>
|
This is still unresolved #1969 (comment) Exactly what steps lead Elasticsearch to not return event IDs for newly created calendar entries? I imagine it's deterministic. We're adding a lot of complexity into this resource to handle missing IDs, that needs to be justified. |
Yeah, it’s deterministic. For normal creates (where we never send event_id), POST always omits it — not intermittently. POST indexes the event, then returns the request payload unchanged. The POST parser doesn’t accept event_id on the request at all, so the in-memory objects never have event_id set before the response is built. The id is assigned at I've confirmed this live on 8.0.1, 8.16.6, and 9.0.8: The POST response has no Terraform needs The pre-create full-calendar snapshot is gone. We are not listing the whole calendar before create anymore. Listing is post-create, and only as a fallback. |
Resolve conflicts with current main (0.15.2): keep ML calendar resources, CompositeIDFromStrForElasticsearch for envelope Read/Delete, and migrate calendar/calendar_event to ElasticsearchResourceOptions and WriteFunc. Co-authored-by: Cursor <cursoragent@cursor.com>
Import used passthrough-only IDs, so malformed import IDs failed during Read with a misleading not-found error instead of Wrong resource ID. Co-authored-by: Cursor <cursoragent@cursor.com>
|
|
||
| if eventID == "" { | ||
| var matches []calendarEventWire | ||
| diags.Append(walkMLCalendarEventPages(ctx, typedClient, calendarID, func(events []calendarEventWire) bool { |
There was a problem hiding this comment.
| diags.Append(walkMLCalendarEventPages(ctx, typedClient, calendarID, func(events []calendarEventWire) bool { | |
| diags.Append(walkMLCalendarEventPagesWithWindow(ctx, typedClient, calendarID, func(events []calendarEventWire) bool { |
We should be able to just look at the event time window here right? Is there any case where we wouldn't have a specific time window to search? It feels like we should be able to remove walkMLCalendarEventPages and only support searching the event window.
There was a problem hiding this comment.
Addressed in 633ce67: post-create ID discovery uses walkMLCalendarEventPagesWithWindow with calendarEventWireWindowRFC3339(planWire).
| sv, vdiags := client.ServerVersion(ctx) | ||
| diags.Append(vdiags...) | ||
| if diags.HasError() { | ||
| return plan, diags | ||
| } | ||
| if sv.LessThan(mlCalendarEventOptionalAPIFieldsMinElasticsearch) { | ||
| diags.AddError( | ||
| "ML calendar event optional scheduling fields not supported", | ||
| fmt.Sprintf("skip_result, skip_model_update, and force_time_shift require Elasticsearch %s or newer; "+ | ||
| "this cluster is %s. Omit these arguments or upgrade Elasticsearch.", | ||
| mlCalendarEventOptionalAPIFieldsMinElasticsearch.String(), sv.String()), | ||
| ) | ||
| return plan, diags | ||
| } |
There was a problem hiding this comment.
This won't work correctly in serverless (it unhelpfully returns 8.11), use EnforceMinVersion instead to check both flavor and version.
There was a problem hiding this comment.
Addressed in 633ce67: optional-field gate uses client.EnforceMinVersion instead of ServerVersion (serverless-safe).
Merge first-slash CompositeIDFromStr parsing for nested resource ids, use ImportStatePassthroughID on import, EnforceMinVersion for optional calendar event fields on serverless, and windowed event listing on create. Co-authored-by: Cursor <cursoragent@cursor.com>
Remove walkMLCalendarEventPages and typedClientCalendarEventsFetcher; use walkMLCalendarEventPagesWithWindow with empty start/end for full scans. Co-authored-by: Cursor <cursoragent@cursor.com>
Resolve CHANGELOG.md conflict: keep main breaking changes and changes, add ML calendar resources under Added. Co-authored-by: Cursor <cursoragent@cursor.com>
Resolve CHANGELOG conflict; keep calendar/calendar_event Added entries with main Fixed section. Reject extra slashes in Kibana synthetics TryReadCompositeID; fix testifylint and modernize lint. Co-authored-by: Cursor <cursoragent@cursor.com>
Resolve conflicts after elastic#1969 merged calendar resources to main: keep calendar_job registration and changelog entry; adopt main's ModifyPlan wiring on calendar and calendar_event resources. Co-authored-by: Cursor <cursoragent@cursor.com>
Summary
elasticstack_elasticsearch_ml_calendarresource for managing ML calendars and job associationselasticstack_elasticsearch_ml_calendar_eventresource for managing individual scheduled events with RFC3339 timesChangelog
Customer impact: enhancement
Summary: Add
elasticstack_elasticsearch_ml_calendarandelasticstack_elasticsearch_ml_calendar_eventresources for managing Elasticsearch ML calendars, scheduled events, and job associations in Terraform.Details
Calendar resource: Creates/deletes via PUT/DELETE calendar API. Updates job associations in-place by diffing
job_idsand calling individual PutCalendarJob/DeleteCalendarJob endpoints.descriptionrequires replacement (PUT is create-only).Calendar event resource: Creates via POST calendar events API, deletes via DELETE. No update API exists — all attributes use RequiresReplace. Server-generated
event_idis discovered by diffing events before/after creation.Test plan
Test*functions ininternal/elasticsearch/ml/calendarandinternal/elasticsearch/ml/calendar_event(models, read/create, wire conversions, validation, paging, composite IDs)TestAcc*functions in those packagesTestAccResourceMLCalendar,TestAccResourceMLCalendarNoJobs,TestAccResourceMLCalendarImport,TestAccResourceMLCalendar_validation_invalidCalendarIDRegex,TestAccResourceMLCalendar_validation_calendarIDTooLong,TestAccResourceMLCalendar_importWrongIDFormatTestAccResourceMLCalendarEvent,TestAccResourceMLCalendarEvent_optionalSchedulingFields,TestAccResourceMLCalendarEventImport,TestAccResourceMLCalendarEvent_validation_endBeforeStart,TestAccResourceMLCalendarEvent_validation_invalidCalendarIDRegex,TestAccResourceMLCalendarEvent_importWrongIDFormatmake buildpasses (including doc generation)Reviewer notes
CompositeIDFromStr/CompositeIDFromStrFwmatches prior releases. Nested segments after the first/apply only toCompositeIDFromStrForElasticsearch/…Fwon the Elasticsearch Plugin Framework envelopeRead/Delete(seeinternal/clients/api_client.goand the unreleased CHANGELOG entry).resource_envelope_test.go: Separate create/update stub functions exist only to assert distinct nil-callback paths (see file header comment).calendar_eventresource:newCalendarEventResourcedocuments why the envelope uses placeholder create callbacks with a customCreateoverride.TestAccResourceMLCalendarEvent_optionalSchedulingFieldsusesversionutils.SkipIfUnsupported; create-time guards increate.goreject optional POST fields on older clusters.Made with Cursor
Note
Add
elasticstack_elasticsearch_ml_calendarandelasticstack_elasticsearch_ml_calendar_eventTerraform resources<cluster_uuid>/<calendar_id>/<event_id>; event IDs are assigned by Elasticsearch and discovered by diffing pre/post-create event lists.start_time,end_time) use RFC3339 strings in Terraform state and are converted to/from epoch milliseconds for the API.📊 Macroscope summarized 5a294b1. 29 files reviewed, 5 issues evaluated, 0 issues filtered, 1 comment posted
(Automatic summaries will resume when PR exits draft mode or review begins).🗂️ Filtered Issues