fix: Use dynamic timezone in now() function for accurate timestamp #18017

Weijun-H · 2025-10-11T09:46:44Z

Which issue does this PR close?

Closes Make now aware of execution timezone #17993

Rationale for this change

DataFusion CLI v50.1.0
> SET TIME ZONE = '+08:00';
0 row(s) fetched. 
Elapsed 0.011 seconds.

> SELECT arrow_typeof(now());
+---------------------------------------+
| arrow_typeof(now())                   |
+---------------------------------------+
| Timestamp(Nanosecond, Some("+08:00")) |
+---------------------------------------+
1 row(s) fetched. 
Elapsed 0.015 seconds.

> SELECT count(1) result FROM (SELECT now() as n) a WHERE n > '2000-01-01'::date;
+--------+
| result |
+--------+
| 1      |
+--------+
1 row(s) fetched. 
Elapsed 0.029 seconds.

What changes are included in this PR?

When the timezone changes, re-register now() function

Are these changes tested?

Are there any user-facing changes?

datafusion/functions/src/datetime/now.rs

Copilot

Pull Request Overview

This PR fixes the now() function to return timestamps with the correct timezone set in the session configuration, instead of always using the default "+00:00" timezone.

Key changes:

Updated now() function to use dynamic timezone from session configuration
Added configuration-aware UDF creation macro for functions that depend on session state
Added test coverage for timezone-aware now() function behavior

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
datafusion/functions/src/macros.rs	Added new macro for creating UDFs with configuration parameters
datafusion/functions/src/datetime/now.rs	Modified NowFunc to accept and use timezone from configuration instead of hardcoded "+00:00"
datafusion/core/src/execution/context/mod.rs	Added logic to re-register now() UDF when timezone configuration changes
datafusion/sqllogictest/test_files/timestamps.slt	Added test to verify now() returns correct timezone type
datafusion/sqllogictest/test_files/dates.slt	Updated expected error message to reflect new default timezone format

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

datafusion/functions/src/datetime/now.rs

Weijun-H · 2025-10-13T19:06:40Z

datafusion/core/src/execution/context/mod.rs

+
+            // Register UDFs that return values based on session configuration
+            // e.g. now() which depends on the time_zone configuration option
+            if variable == "datafusion.execution.time_zone" {
+                let config_options = self.state.read().config().options().clone();
+                let now_udf = {
+                    // recreate the function so it captures the new time zone
+                    make_udf_function_with_config!(NowFunc, now, &ConfigOptions);
+                    now(&config_options)
+                };
+                self.state.write().register_udf(now_udf)?;
+            }


We could refactor this logic after these issue resolved:

Make current_date aware of execution timezone #17995

Make to_timestamp aware of execution timezone #17998

Make current_date aware of execution timezone #17995

I think the above approach will handle all future udf changes that follow this pattern

kosiew · 2025-10-14T09:24:51Z

Re-registering now() inside set_config_option covers interactive SET commands, but sessions that are constructed differently eg with a pre-configured SessionConfig (e.g. via builder APIs or server defaults) still get the default NowFunc::new() that ignores the configured timezone.

eg.

❯ DATAFUSION_EXECUTION_TIME_ZONE='+05:45' cargo run --bin datafusion-cli -- -c "SELECT arrow_typeof(now());"

DataFusion CLI v50.1.0
+------------------------------------+
| arrow_typeof(now())                |
+------------------------------------+
| Timestamp(Nanosecond, Some("+00")) |
+------------------------------------+
1 row(s) fetched.
Elapsed 0.092 seconds.

❯ cargo run --bin datafusion-cli -- -c "SET datafusion.execution.time_zone = '+05:45'; SELECT arrow_typeof(now());"
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.48s
     Running `target/debug/datafusion-cli -c 'SET datafusion.execution.time_zone = '\''+05:45'\''; SELECT arrow_typeof(now());'`
DataFusion CLI v50.1.0
0 row(s) fetched.
Elapsed 0.034 seconds.

+---------------------------------------+
| arrow_typeof(now())                   |
+---------------------------------------+
| Timestamp(Nanosecond, Some("+05:45")) |
+---------------------------------------+
1 row(s) fetched.
Elapsed 0.040 seconds.

datafusion/core/src/execution/context/mod.rs

datafusion/expr/src/udf.rs

Weijun-H · 2025-10-14T20:07:38Z

Thanks @Omega359 and @kosiew for reviewing. I think this pr is ready for review.

╰─ DATAFUSION_EXECUTION_TIME_ZONE='+05:45' cargo run --bin datafusion-cli -- -c "SELECT arrow_typeof(now());"                                           ─╯
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.53s
     Running `target/debug/datafusion-cli -c 'SELECT arrow_typeof(now());'`
DataFusion CLI v50.1.0
+---------------------------------------+
| arrow_typeof(now())                   |
+---------------------------------------+
| Timestamp(Nanosecond, Some("+05:45")) |
+---------------------------------------+
1 row(s) fetched. 
Elapsed 0.011 seconds.

Co-authored-by: Andrew Lamb <[email protected]>

Weijun-H · 2025-10-20T07:47:56Z

@kosiew @Jefffrey @Omega359 @alamb

Any additional feedback on this PR? If there are no blocking issues, I'd like to merge soon.

kosiew

LGTM except for the deprecated NowFunc::new(), still seeding the timezone with "+00"

kosiew · 2025-10-20T12:51:07Z

datafusion/functions/src/datetime/now.rs

        Self {
            signature: Signature::nullary(Volatility::Stable),
            aliases: vec!["current_timestamp".to_string()],
+            timezone: Some(Arc::from("+00")),


NowFunc::new(), though deprecated seeds the timezone with "+00" (missing minutes) while the canonical UTC offset remains "+00:00" in ConfigOptions::default().
Existing callers of the deprecated constructor will therefore get a different datatype/ScalarValue than before this change.

Weijun-H · 2025-10-22T09:13:15Z

Thanks again @kosiew @Jefffrey @Omega359 @alamb

Omega359 · 2025-10-22T12:58:22Z

LGTM except for the deprecated NowFunc::new(), still seeding the timezone with "+00"

Filed as #18219

## Which issue does this PR close? - Relates #18062 - Relates #18065 ## Rationale for this change We disabled these tests because CI was failing on main. The test: `current_date() = cast(now() as date)` was added in #18034 requires `now` to use configured timezone, but it is only available after #18017. Since #18017 has been merged, these tests should be enable. ## What changes are included in this PR? This reverts commit a65a2cb. ## Are these changes tested? Yes. ## Are there any user-facing changes? No.

…pache#18017) ## Which issue does this PR close?  - Closes apache#17993 ## Rationale for this change ``` DataFusion CLI v50.1.0 > SET TIME ZONE = '+08:00'; 0 row(s) fetched. Elapsed 0.011 seconds. > SELECT arrow_typeof(now()); +---------------------------------------+ | arrow_typeof(now()) | +---------------------------------------+ | Timestamp(Nanosecond, Some("+08:00")) | +---------------------------------------+ 1 row(s) fetched. Elapsed 0.015 seconds. > SELECT count(1) result FROM (SELECT now() as n) a WHERE n > '2000-01-01'::date; +--------+ | result | +--------+ | 1 | +--------+ 1 row(s) fetched. Elapsed 0.029 seconds. ```  ## What changes are included in this PR? When the timezone changes, re-register `now()` function  ## Are these changes tested?  ## Are there any user-facing changes?   --------- Co-authored-by: Andrew Lamb <[email protected]>

## Which issue does this PR close? - Relates apache#18062 - Relates apache#18065 ## Rationale for this change We disabled these tests because CI was failing on main. The test: `current_date() = cast(now() as date)` was added in apache#18034 requires `now` to use configured timezone, but it is only available after apache#18017. Since apache#18017 has been merged, these tests should be enable. ## What changes are included in this PR? This reverts commit a65a2cb. ## Are these changes tested? Yes. ## Are there any user-facing changes? No.

Weijun-H changed the title ~~fix: Use dynamic timezone in now() function for accurate timestamp~~ WIP fix: Use dynamic timezone in now() function for accurate timestamp Oct 11, 2025

github-actions bot added the functions Changes to functions implementation label Oct 11, 2025

Weijun-H force-pushed the 17993-make-now-aware-time-zone branch from 8db266f to 33732c5 Compare October 11, 2025 12:25

github-actions bot added the sqllogictest SQL Logic Tests (.slt) label Oct 11, 2025

Weijun-H changed the title ~~WIP fix: Use dynamic timezone in now() function for accurate timestamp~~ fix: Use dynamic timezone in now() function for accurate timestamp Oct 11, 2025

Weijun-H marked this pull request as draft October 11, 2025 12:28

Omega359 reviewed Oct 11, 2025

View reviewed changes

datafusion/functions/src/datetime/now.rs Outdated Show resolved Hide resolved

Weijun-H force-pushed the 17993-make-now-aware-time-zone branch 2 times, most recently from 70ed548 to b81ebf6 Compare October 12, 2025 10:03

github-actions bot added the core Core DataFusion crate label Oct 13, 2025

Weijun-H force-pushed the 17993-make-now-aware-time-zone branch from f7ba4e5 to 9f1b55f Compare October 13, 2025 19:00

Weijun-H requested a review from Copilot October 13, 2025 19:01

Copilot AI reviewed Oct 13, 2025

View reviewed changes

datafusion/functions/src/datetime/now.rs Show resolved Hide resolved

datafusion/functions/src/datetime/now.rs Outdated Show resolved Hide resolved

Weijun-H marked this pull request as ready for review October 13, 2025 19:05

Weijun-H commented Oct 13, 2025

View reviewed changes

Weijun-H marked this pull request as draft October 13, 2025 20:00

Weijun-H force-pushed the 17993-make-now-aware-time-zone branch from 34ad88f to e7a6ab9 Compare October 13, 2025 20:11

Weijun-H marked this pull request as ready for review October 13, 2025 20:11

Weijun-H force-pushed the 17993-make-now-aware-time-zone branch from 8a1de2c to ed73c86 Compare October 13, 2025 21:30

Weijun-H requested a review from Omega359 October 14, 2025 07:25

kosiew mentioned this pull request Oct 14, 2025

Respect execution timezone in to_timestamp and related functions #18025

Open

github-actions bot added the logical-expr Logical plan and expressions label Oct 14, 2025

Omega359 reviewed Oct 14, 2025

View reviewed changes

datafusion/core/src/execution/context/mod.rs Outdated Show resolved Hide resolved

Omega359 reviewed Oct 14, 2025

View reviewed changes

datafusion/expr/src/udf.rs Outdated Show resolved Hide resolved

Weijun-H force-pushed the 17993-make-now-aware-time-zone branch 3 times, most recently from 1310d0f to 179607a Compare October 14, 2025 20:52

dqkqd mentioned this pull request Oct 14, 2025

CI is failing on main / current_date() = cast(now() as date); #18062

Closed

Weijun-H and others added 15 commits October 20, 2025 10:46

feat: add configurable UDF for now() function with timezone support

aecb2d1

chore

83eab48

chore: clippy

e3425a7

chore

68445af

chore: fix test

5752326

fix: initialize the function when building based on the config option

eac80ba

refactor: update UDF handling to support runtime configuration changes

09535a2

chore

31c4656

chore: Update tests

c6dccd3

chore

ae0016b

chore: clippy

ca2fe04

refactor: update NowFunc default implementation to use new_with_config

607e30e

refactor: simplify logic and clarify documentation

fa6f71e

Update datafusion/expr/src/udf.rs

0bbb2fe

Co-authored-by: Andrew Lamb <[email protected]>

chore: fmt

4e8a0c3

Weijun-H force-pushed the 17993-make-now-aware-time-zone branch from f9fbf67 to 4e8a0c3 Compare October 20, 2025 07:46

kosiew approved these changes Oct 20, 2025

View reviewed changes

alamb mentioned this pull request Oct 21, 2025

Default time_zone config to None #18204

Open

Weijun-H added this pull request to the merge queue Oct 22, 2025

Merged via the queue into apache:main with commit b7a10ad Oct 22, 2025
55 of 56 checks passed

Omega359 mentioned this pull request Oct 22, 2025

NowFunc::new() seeds the timezone with "+00" while the canonical UTC offset remains "+00:00" in ConfigOptions::default(). #18219

Closed

alamb mentioned this pull request Oct 22, 2025

[EPIC] Complete datafusion-spark Spark Compatible Functions #15914

Open

19 tasks

dqkqd mentioned this pull request Oct 23, 2025

Revert "chore: revert tests (#18065)" #18255

Merged

fix: Use dynamic timezone in now() function for accurate timestamp #18017

fix: Use dynamic timezone in now() function for accurate timestamp #18017

Uh oh!

Conversation

Weijun-H commented Oct 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Weijun-H Oct 13, 2025

Choose a reason for hiding this comment

Uh oh!

Omega359 Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

kosiew commented Oct 14, 2025

Uh oh!

Uh oh!

Uh oh!

Weijun-H commented Oct 14, 2025

Uh oh!

Weijun-H commented Oct 20, 2025

Uh oh!

kosiew left a comment

Choose a reason for hiding this comment

Uh oh!

kosiew Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Weijun-H commented Oct 22, 2025

Uh oh!

Omega359 commented Oct 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Weijun-H commented Oct 11, 2025 •

edited

Loading