Skip to content

[BUG] DateType writes use JVM default timezone midnight, shifting the calendar day on non UTC JVMs #797

Description

@lawofcycles

What is the bug?

When the Spark connector writes a Spark DateType value to OpenSearch, DataFrameValueWriter serializes it to epoch millis via value.asInstanceOf[Date].getTime(). A java.sql.Date represents midnight in the JVM default timezone, so on a non UTC JVM this writes local midnight instead of UTC midnight and the stored value can land on a different calendar day. The same code is present in the sql-30, sql-35, and sql-40 modules.

How can one reproduce the bug?

  1. Set the JVM default timezone to a non UTC zone such as Asia/Tokyo (UTC+9).
  2. Create a DataFrame with a DateType column holding 2023-07-22.
  3. Write it to an OpenSearch index whose field is a date.
  4. Observe the stored value is 2023-07-21T15:00:00Z, so it reads back as the previous day.

What is the expected behavior?

A DateType is a timezone independent calendar date, so it should be stored as UTC start of day (2023-07-22T00:00:00Z). A tz independent conversion such as d.toLocalDate.atStartOfDay(java.time.ZoneOffset.UTC).toInstant.toEpochMilli produces the correct value.

What is your host/environment?

Any non UTC JVM timezone, for example Asia/Tokyo. Reproducible on current main across all Spark modules.

Do you have any screenshots?

NA

Do you have any additional context?

NA

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions