What is the bug?
When the Spark connector writes a Spark DateType value to OpenSearch, DataFrameValueWriter serializes it to epoch millis via value.asInstanceOf[Date].getTime(). A java.sql.Date represents midnight in the JVM default timezone, so on a non UTC JVM this writes local midnight instead of UTC midnight and the stored value can land on a different calendar day. The same code is present in the sql-30, sql-35, and sql-40 modules.
How can one reproduce the bug?
- Set the JVM default timezone to a non UTC zone such as Asia/Tokyo (UTC+9).
- Create a DataFrame with a
DateType column holding 2023-07-22.
- Write it to an OpenSearch index whose field is a date.
- Observe the stored value is
2023-07-21T15:00:00Z, so it reads back as the previous day.
What is the expected behavior?
A DateType is a timezone independent calendar date, so it should be stored as UTC start of day (2023-07-22T00:00:00Z). A tz independent conversion such as d.toLocalDate.atStartOfDay(java.time.ZoneOffset.UTC).toInstant.toEpochMilli produces the correct value.
What is your host/environment?
Any non UTC JVM timezone, for example Asia/Tokyo. Reproducible on current main across all Spark modules.
Do you have any screenshots?
NA
Do you have any additional context?
NA
What is the bug?
When the Spark connector writes a Spark
DateTypevalue to OpenSearch,DataFrameValueWriterserializes it to epoch millis viavalue.asInstanceOf[Date].getTime(). Ajava.sql.Daterepresents midnight in the JVM default timezone, so on a non UTC JVM this writes local midnight instead of UTC midnight and the stored value can land on a different calendar day. The same code is present in the sql-30, sql-35, and sql-40 modules.How can one reproduce the bug?
DateTypecolumn holding2023-07-22.2023-07-21T15:00:00Z, so it reads back as the previous day.What is the expected behavior?
A
DateTypeis a timezone independent calendar date, so it should be stored as UTC start of day (2023-07-22T00:00:00Z). A tz independent conversion such asd.toLocalDate.atStartOfDay(java.time.ZoneOffset.UTC).toInstant.toEpochMilliproduces the correct value.What is your host/environment?
Any non UTC JVM timezone, for example Asia/Tokyo. Reproducible on current main across all Spark modules.
Do you have any screenshots?
NA
Do you have any additional context?
NA