Skip to content

[GroupBy] Add spark conf flag to resolve ChrononRunDs from queryRange - #1113

Open
yogeshpv wants to merge 2 commits into
airbnb:mainfrom
yogeshpv:main
Open

[GroupBy] Add spark conf flag to resolve ChrononRunDs from queryRange#1113
yogeshpv wants to merge 2 commits into
airbnb:mainfrom
yogeshpv:main

Conversation

@yogeshpv

Copy link
Copy Markdown
Contributor

Summary

Add opt-in Spark conf flag to resolve ChrononRunDs macro from queryRange instead of intersectedRange in GroupBy source query rendering.

Why / Goal

Currently, the ChrononRunDs macro in renderDataSourceQuery is resolved using intersectedRange.start, which requires the intersected range to be defined and single-day. This works for most cases but is limiting when callers need the macro to reflect the queryRange directly (e.g., when the query date and the scanned data range diverge).
This change introduces a Spark conf flag spark.chronon.group_by.use_query_range_for_run_ds (default false) that, when enabled, resolves ChrononRunDs from queryRange.start (with SQL string quotes) instead of intersectedRange.start. The existing default behavior is fully preserved when the flag is not set.

Test Plan

  • Added Unit Tests — testChrononRunDsWithQueryRangeFlag in JoinTest.scala validates the flag-enabled path renders queryRange.start into the query
  • Covered by existing CI — existing testSourceQueryRender continues to cover the default (flag-off) path
  • Integration tested

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants