(#34659) - Add Timezone Configuration to Watcher (#117033)

* Add timezone support to Cron objects

* Add timezone support to CronnableSchedule

* XContent change to support parsing and display of TimeZone fields on schedules

* Case insensitive timezone parsing

* Doc changes

* YAML REST tests

* Equals, toString and HashCode now include timezone

* Additional random testing for DST transitions

* Migrate Cron class to use wrapped LocalDateTime

The algorithm depends on some quirks of calendar but LocalDateTime
correctly ignores DST during calculations so this uses a LocalDateTime
with a wrapper to emulate some of Calendar's behaviours that the Cron
algorithm depends on

* Additional documentation to explain discontinuity event behaviour

* Remove redundant conversions from ZoneId to TimeZone following move to LocalDateTime

* Add documentation warning that manual clock changes will cause unpredictable watch execution

* Update docs/reference/watcher/trigger/schedule.asciidoc

Co-authored-by: Lee Hinman <[email protected]>

---------

Co-authored-by: Lee Hinman <[email protected]>

Author: lukewhiting

docs/reference/watcher/example-watches/watching-time-series-data.asciidoc

@@ -62,20 +62,20 @@ contain the words "error" or "problem".
 
 To set up the watch:
 
-. Define the watch trigger--a daily schedule that runs at 12:00 UTC:
+. Define the watch trigger--a daily schedule that runs at 12:00 Australian Eastern Standard Time (UTC+10:00):
 +
 [source,js]
 --------------------------------------------------
 "trigger" : {
   "schedule" : {
+    "timezone": "Australia/Brisbane",
     "daily" : { "at" : "12:00" }
   }
 }
 --------------------------------------------------
 +
-NOTE: In {watcher}, you specify times in UTC time. Don't forget to do the
-      conversion from your local time so the schedule triggers at the time
-      you intend.
+NOTE: In {watcher}, if the timezone is omitted then schedules default to UTC. `timezone` can be specified either
+as a +/-HH:mm offset from UTC or as a timezone name from the machines local IANA Time Zone Database.
 
 . Define the watch input--a search that uses a filter to constrain the results
   to the past day.

docs/reference/watcher/trigger/schedule.asciidoc

@@ -6,12 +6,42 @@
 ++++
 
 Schedule <<trigger,triggers>> define when the watch execution should start based
-on date and time. All times are specified in UTC time.
+on date and time. All times are in UTC time unless a timezone is explicitly specified
+in the schedule.
 
 {watcher} uses the system clock to determine the current time. To ensure schedules
 are triggered when expected, you should synchronize the clocks of all nodes in the
 cluster using a time service such as http://www.ntp.org/[NTP].
 
+NOTE: {watcher} can't correct for manual adjustments to the system clock. Be aware when making
+such changes that watch execution may be affected with watches being skipped or repeated if the
+adjustment covers their target execution time. This applies to changes made via NTP as well.
+
+When specifying a timezone for a watch, keep in mind the effect daylight savings time
+transitions may have on the schedule, especially if the watch is scheduled to run
+during the transition. Here's how {watcher} handles watches scheduled during discontinuities:
+
+==== Gap Transitions
+These occur when the clock moves forward, such as when daylight savings time starts
+and cause certain hours or minutes to be skipped. If your watch is scheduled to run
+during a gap transition, the watch is executed at the same time as before the transition.
+
+Example: If a watch is scheduled to run daily at 1:30AM in the `Europe/London` time zone and
+the clock moves forward one hour from 1:00AM (GMT+0) to 2:00AM (GMT+1), the watch is executed
+at 2:30AM (GMT+1) which would have been 1:30AM before the transition. Subsequent executions
+happen at 1:30AM (GMT+1).
+
+==== Overlap Transitions
+These occur when the clock moves backward, such as when daylight savings time ends
+and cause certain hours or minutes to be repeated. If your watch is scheduled to run
+during an overlap transition, only the first occurrence of the time causes to the watch
+to execute with the second being skipped.
+
+Example: If a watch is scheduled to run at 1:30 AM and the clock moves backward one hour
+from 2:00AM to 1:00AM, the watch is executed at 1:30AM and the second occurrence after the
+change is skipped.
+
+=== Throttling
 Keep in mind that the throttle period can affect when a watch is actually executed.
 The default throttle period is five seconds (5000 ms). If you configure a schedule
 that's more frequent than the throttle period, the throttle period overrides the
@@ -20,6 +50,7 @@ and set the schedule to every 10 seconds, the watch is executed no more than
 once per minute. For more information about throttling, see
 <<actions-ack-throttle>>.
 
+=== Schedule Types
 {watcher} provides several types of schedule triggers:
 
 * <<schedule-hourly, `hourly`>>

docs/reference/watcher/trigger/schedule/cron.asciidoc

@@ -5,14 +5,14 @@
 ++++
 
 
-Defines a <<trigger-schedule, `schedule`>> using a <<api-cron-expressions, cron expression>> 
+Defines a <<trigger-schedule, `schedule`>> using a <<api-cron-expressions, cron expression>>
 that specifiues when to execute a watch.
 
 
-TIP:  While cron expressions are powerful, a regularly occurring schedule 
-is easier to configure with the other schedule types. 
-If you must use a cron schedule, make sure you verify it with 
-<<elasticsearch-croneval, `elasticsearch-croneval`>> . 
+TIP:  While cron expressions are powerful, a regularly occurring schedule
+is easier to configure with the other schedule types.
+If you must use a cron schedule, make sure you verify it with
+<<elasticsearch-croneval, `elasticsearch-croneval`>> .
 
 
 ===== Configure a cron schedule with one time
@@ -60,16 +60,40 @@ minute during the weekend:
 --------------------------------------------------
 // NOTCONSOLE
 
+[[configue_cron_time-zone]]
+==== Use a different time zone for a cron schedule
+By default, cron expressions are evaluated in the UTC time zone. To use a different time zone,
+you can specify the `timezone` parameter in the schedule. For example, the following
+`cron` schedule triggers at 6:00 AM and 6:00 PM during weekends in the `America/Los_Angeles` time zone:
+
+
+[source,js]
+--------------------------------------------------
+{
+  ...
+  "trigger" : {
+    "schedule" : {
+      "timezone" : "America/Los_Angeles",
+      "cron" : [
+        "0 6,18 * * * SAT-SUN",
+      ]
+    }
+  }
+  ...
+}
+--------------------------------------------------
+// NOTCONSOLE
+
 [[croneval]]
 ===== Use croneval to validate cron expressions
 
-{es} provides a <<elasticsearch-croneval, `elasticsearch-croneval`>> command line tool 
-in the `$ES_HOME/bin` directory that you can use to check that your cron expressions 
+{es} provides a <<elasticsearch-croneval, `elasticsearch-croneval`>> command line tool
+in the `$ES_HOME/bin` directory that you can use to check that your cron expressions
 are valid and produce the expected results.
 
-To validate a cron expression, pass it in as a parameter to `elasticsearch-croneval`:	
+To validate a cron expression, pass it in as a parameter to `elasticsearch-croneval`:
 
 [source,bash]
 --------------------------------------------------
 bin/elasticsearch-croneval "0 0/1 * * * ?"
---------------------------------------------------	
+--------------------------------------------------

docs/reference/watcher/trigger/schedule/daily.asciidoc

@@ -97,3 +97,27 @@ or minutes as an array. For example, following `daily` schedule triggers at
 }
 --------------------------------------------------
 // NOTCONSOLE
+
+[[specifying-time-zone-for-daily-schedule]]
+===== Specifying a time zone for a daily schedule
+By default, daily schedules are evaluated in the UTC time zone. To use a different time zone,
+you can specify the `timezone` parameter in the schedule. For example, the following
+`daily` schedule triggers at 6:00 AM and 6:00 PM in the `Pacific/Galapagos` time zone:
+
+[source,js]
+--------------------------------------------------
+{
+  "trigger" : {
+    "schedule" : {
+      "timezone" : "Pacific/Galapagos",
+      "daily" : {
+        "at" : {
+          "hour" : [ 6, 18 ],
+          "minute" : 0
+        }
+      }
+    }
+  }
+}
+--------------------------------------------------
+// NOTCONSOLE

docs/reference/watcher/trigger/schedule/monthly.asciidoc

@@ -74,4 +74,26 @@ schedule triggers at 12:00 AM and 12:00 PM on the 10th and 20th of each month.
   }
 }
 --------------------------------------------------
-// NOTCONSOLE
+// NOTCONSOLE
+
+==== Configuring time zones for monthly schedules
+By default, monthly schedules are evaluated in the UTC time zone. To use a different
+time zone, you can specify the `timezone` parameter in the schedule. For example,
+the following `monthly` schedule triggers at 6:00 AM and 6:00 PM on the 15th of each month in
+the `Asia/Tokyo` time zone:
+
+[source,js]
+--------------------------------------------------
+{
+  "trigger" : {
+    "schedule" : {
+      "timezone" : "Asia/Tokyo",
+      "monthly" : {
+        "on" : [ 15 ],
+        "at" : [ 6:00, 18:00 ]
+      }
+    }
+  }
+}
+--------------------------------------------------
+// NOTCONSOLE

docs/reference/watcher/trigger/schedule/weekly.asciidoc

@@ -79,4 +79,26 @@ Alternatively, you can specify days and times in an object that has `on` and
   }
 }
 --------------------------------------------------
-// NOTCONSOLE
+// NOTCONSOLE
+
+==== Use a different time zone for a weekly schedule
+By default, weekly schedules are evaluated in the UTC time zone. To use a different time zone,
+you can specify the `timezone` parameter in the schedule. For example, the following
+`weekly` schedule triggers at 6:00 AM and 6:00 PM on Tuesdays and Fridays in the
+`America/Buenos_Aires` time zone:
+
+[source,js]
+--------------------------------------------------
+{
+  "trigger" : {
+    "schedule" : {
+      "timezone" : "America/Buenos_Aires",
+      "weekly" : {
+        "on" : [ "tuesday", "friday" ],
+        "at" : [ "6:00", "18:00" ]
+      }
+    }
+  }
+}
+--------------------------------------------------
+// NOTCONSOLE

docs/reference/watcher/trigger/schedule/yearly.asciidoc

@@ -88,3 +88,26 @@ January 20th, December 10th, and December 20th.
 }
 --------------------------------------------------
 // NOTCONSOLE
+
+==== Configuring a yearly schedule with a different time zone
+By default, the `yearly` schedule is evaluated in the UTC time zone. To use a
+different time zone, you can specify the `timezone` parameter in the schedule.
+For example, the following `yearly` schedule triggers at 3:30 PM and 8:30 PM
+on June 4th in the `Antarctica/Troll` time zone:
+
+[source,js]
+--------------------------------------------------
+{
+  "trigger" : {
+    "schedule" : {
+      "timezone" : "Antarctica/Troll",
+      "yearly" : {
+          "in" : "june",
+          "on" : 4,
+          "at" : [ 15:30, 20:30 ]
+      }
+    }
+  }
+}
+--------------------------------------------------
+// NOTCONSOLE