Optimize FIRST_VALUE/LAST_VALUE with type-specialized argmin/argmax #65
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…#17985) * refactor: rename macros full clear meaning * refactor: cleanup and simplify regexp match macros * refactor: update error messages
…he#18046) * Impl gather_filters_for_pushdown for CoalescePartitionsExec * add tests
* Add test * Use ROWS instead of RANGE * Fix a test * progress * window.slt like master * passing existing tests * Break out window limit tests * LimitEffect * fix a bug * repartitions * refactor * refactor * fmt * remove casual * two phased approach * refactor into context * refactor * refactor * refactor * remove comments * remove deps * Fix NthValue * aggregates * ranking functions * More tests * Max lead test * More tests, JIC * More tests, JIC * Notes * Notes--
* Add test * Fix fmt and test
* update governance page * prettier
* Feat: Make current_date aware of execution timezone. * CI Fixes: Rectify assertion error of slt and update scalar_functions.md * CI Fixes: Comment out flaky test involving now(). * CI Fixes: Resolve slt error. * Feat: Add helper function to calculate current_date and fix tests. * Chore: Refactor timezone conversion logic.
Bumps [actions/setup-node](https://github.com/actions/setup-node) from 5.0.0 to 6.0.0. - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](actions/setup-node@a0853c2...2028fbc) --- updated-dependencies: - dependency-name: actions/setup-node dependency-version: 6.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [sysinfo](https://github.com/GuillaumeGomez/sysinfo) from 0.37.1 to 0.37.2. - [Changelog](https://github.com/GuillaumeGomez/sysinfo/blob/master/CHANGELOG.md) - [Commits](GuillaumeGomez/sysinfo@v0.37.1...v0.37.2) --- updated-dependencies: - dependency-name: sysinfo dependency-version: 0.37.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* feat(spark): implement Spark elt function * feat(spark): implement Spark elt function license * feat(spark): implement Spark elt function license complettly * test slt * doc * wip * change error * fmt * some commts * some changes * coerce basic <- udf * coerce basic drop 32 * change n,k,j names * first value int64 or casteable * change revision * change revision test * change revision test fmt
…he#17966) * FileScanConfig: preserve schema metadata across serde boundary * add roundtrip_physical_plan test with filescanconfig schema metadata * lint
Filters are safe to be pushed down, so we can override the default behavior here. Signed-off-by: Alfonso Subiotto Marques <[email protected]>
* Adds summary output to CLI instrumented object stores - Adds a `RequestSummary` type for the instrumented object store to display summary statistics about instrumented requests - Adds a generic Stats type to track the statistics for the summary - Adds tests for the new code - Adds a basic summary output to the user-facing display when profiling is enabled - Adds docs for new and newly exported public items * - Updates integration test and validation snapshot
* Impl spark bit not function * Impl spark bit not function * Fix format * Fix format * Fix Clippy warnings * Rename func * Add .slt tests * Fix fmt --------- Co-authored-by: Kazantsev Maksim <[email protected]>
* chore: revert tests * chore: revert tests
…evaluation (apache#17973) * apache#17972 Restore case expr/expr optimisation while ensuring lazy evaluation * Avoid calling `PhysicalExpr::evaluate` from `PhysicalExpr::evaluate_selection` for empty selections. * Make `PhysicalExpr::evaluate_selection` correctly handle empty input sets and all false filters * Reoragnize code to avoid scatter codepath when using `evaluate` fast path. * Clarify comments in case * Move null handling after true count check. * Tweaking comments * Add unit tests to help define the boundary case behaviour of evaluate_selection * Code polishing - Add extra comments - Use match for the scatter paragraph - Validate that the size of selection and batch match * Fix clippy errors * Add additional case SLTs --------- Co-authored-by: Andrew Lamb <[email protected]>
…n array (apache#18048) * chore: Use an enum to express the different kinds of nullability in an array Follow-up of apache#17726 (review) * Use the Nulls enum also in .../multi_group_by/primitive.rs * Use the Nulls enum in .../multi_group_by/bytes[_view].rs as well
…che#18069) Bumps [taiki-e/install-action](https://github.com/taiki-e/install-action) from 2.62.28 to 2.62.29. - [Release notes](https://github.com/taiki-e/install-action/releases) - [Changelog](https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md) - [Commits](taiki-e/install-action@e7ef886...5b5de1b) --- updated-dependencies: - dependency-name: taiki-e/install-action dependency-version: 2.62.29 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…sed (apache#17958) * fix: Prune partitions when no filters are defined * fix: Formatting * chore: Cargo fmt * chore: Clippy
…pache#17906) * Check-in NestedLoopJoinProjectionPushDown * Update Cargo.lock * Add some comments * Update slts that are affected by the nl-join-projection-push-down * please lints * Move code into projection_pushdown.rs * Remove explicit coalesce batches * Docs
* chore: Extend backtrace coverage * fmt * part2 * feedback * clippy * feat: support Spark `concat` * clippy * comments * test * doc
* Add independent configs for topk/join dynamic filter * fix ci * update doc * fix typo
- Adds the ability for a user to choose a summary only output for an instrumented object store when using the CLI - The existing "enabled" setting that displays both a summary and a detailed usage for each object store call has been renamed to `Trace` to improve clarity - Adds additional test cases for summary only and modifies existing tests to use trace - Updates user guide docs to reflect the CLI flag and command line changes
* fix: Improve null handling in array_to_string function * chore
## Which issue does this PR close? - Closes apache#18084 ## Rationale for this change Some of the extended tests are failing because we have fixed case conditional evaluation and queries that (incorrectly) previously did not pass are now. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? Update datafusion-testing pin ## Are these changes tested? I tested locally with: ```shell INCLUDE_SQLITE=true cargo test --profile release-nonlto --test sqllogictests ``` ## Are there any user-facing changes? No
…che#18094) Bumps [taiki-e/install-action](https://github.com/taiki-e/install-action) from 2.62.29 to 2.62.31. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/taiki-e/install-action/releases">taiki-e/install-action's releases</a>.</em></p> <blockquote> <h2>2.62.31</h2> <ul> <li> <p>Update <code>protoc@latest</code> to 3.33.0.</p> </li> <li> <p>Update <code>uv@latest</code> to 0.9.3.</p> </li> <li> <p>Update <code>syft@latest</code> to 1.34.1.</p> </li> <li> <p>Update <code>mise@latest</code> to 2025.10.9.</p> </li> <li> <p>Update <code>cargo-shear@latest</code> to 1.6.0.</p> </li> </ul> <h2>2.62.30</h2> <ul> <li> <p>Update <code>vacuum@latest</code> to 0.18.6.</p> </li> <li> <p>Update <code>zizmor@latest</code> to 1.15.2.</p> </li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md">taiki-e/install-action's changelog</a>.</em></p> <blockquote> <h1>Changelog</h1> <p>All notable changes to this project will be documented in this file.</p> <p>This project adheres to <a href="https://semver.org">Semantic Versioning</a>.</p> <!-- raw HTML omitted --> <h2>[Unreleased]</h2> <h2>[2.62.31] - 2025-10-16</h2> <ul> <li> <p>Update <code>protoc@latest</code> to 3.33.0.</p> </li> <li> <p>Update <code>uv@latest</code> to 0.9.3.</p> </li> <li> <p>Update <code>syft@latest</code> to 1.34.1.</p> </li> <li> <p>Update <code>mise@latest</code> to 2025.10.9.</p> </li> <li> <p>Update <code>cargo-shear@latest</code> to 1.6.0.</p> </li> </ul> <h2>[2.62.30] - 2025-10-15</h2> <ul> <li> <p>Update <code>vacuum@latest</code> to 0.18.6.</p> </li> <li> <p>Update <code>zizmor@latest</code> to 1.15.2.</p> </li> </ul> <h2>[2.62.29] - 2025-10-14</h2> <ul> <li> <p>Update <code>zizmor@latest</code> to 1.15.1.</p> </li> <li> <p>Update <code>cargo-nextest@latest</code> to 0.9.106.</p> </li> <li> <p>Update <code>mise@latest</code> to 2025.10.8.</p> </li> <li> <p>Update <code>ubi@latest</code> to 0.8.1.</p> </li> </ul> <h2>[2.62.28] - 2025-10-11</h2> <ul> <li> <p>Update <code>release-plz@latest</code> to 0.3.148.</p> </li> <li> <p>Update <code>cargo-sort@latest</code> to 2.0.2.</p> </li> <li> <p>Update <code>cargo-binstall@latest</code> to 1.15.7.</p> </li> <li> <p>Update <code>uv@latest</code> to 0.9.2.</p> </li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/taiki-e/install-action/commit/0005e0116e92d8489d8d96fbff83f061c79ba95a"><code>0005e01</code></a> Release 2.62.31</li> <li><a href="https://github.com/taiki-e/install-action/commit/6936d999d90424ed013e4f325d91e14d7ddba27f"><code>6936d99</code></a> Update <code>protoc@latest</code> to 3.33.0</li> <li><a href="https://github.com/taiki-e/install-action/commit/ac7ad6efa1b1bb919bcaa357eb1873f328ee07f7"><code>ac7ad6e</code></a> Update <code>uv@latest</code> to 0.9.3</li> <li><a href="https://github.com/taiki-e/install-action/commit/005833aaf18c1621513995406c3bc0397747afc2"><code>005833a</code></a> Update <code>syft@latest</code> to 1.34.1</li> <li><a href="https://github.com/taiki-e/install-action/commit/2b32ff6f3dc99bc9fa6647cbc9f7da71cf979b65"><code>2b32ff6</code></a> Update <code>mise@latest</code> to 2025.10.9</li> <li><a href="https://github.com/taiki-e/install-action/commit/74c0274864f156f487aee04623a20b315fb2125a"><code>74c0274</code></a> Update <code>cargo-shear@latest</code> to 1.6.0</li> <li><a href="https://github.com/taiki-e/install-action/commit/f13d8e15c52b25c79b608d399cc802adc73d83da"><code>f13d8e1</code></a> Release 2.62.30</li> <li><a href="https://github.com/taiki-e/install-action/commit/1034dc55996706645239db97d3ea04f42a708f22"><code>1034dc5</code></a> Update <code>vacuum@latest</code> to 0.18.6</li> <li><a href="https://github.com/taiki-e/install-action/commit/55b5d509b8761e9696e1cfec0d6f66f0655e8fff"><code>55b5d50</code></a> Update <code>zizmor@latest</code> to 1.15.2</li> <li>See full diff in <a href="https://github.com/taiki-e/install-action/compare/5b5de1b4da26ad411330c0454bdd72929bfcbeb2...0005e0116e92d8489d8d96fbff83f061c79ba95a">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes #. Related to apache#18084 ## Rationale for this change Run extended suite on PRs for critical areas, to avoid post merge bugfixing <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Andrew Lamb <[email protected]>
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> Part of apache#14763 and apache#14760 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Current `log()` signature has some drawbacks: https://github.com/apache/datafusion/blob/a5eb9121ccf802dda547897155403b08a4fbf774/datafusion/functions/src/math/log.rs#L78-L105 - A bit nasty to look at: mixes numeric with exact float/int with exact decimal (of exact precision and scale) - Can't accommodate arbitrary decimals of any precision/scale (this is true for other functions too) Aim of this PR is to refactor it to use the coercion API, uplifting the API where necessary to make this possible. This simplifies the signature in code, whilst not losing flexibility. Also other minor refactors are included to log. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> New `TypeSignatureClass` variants: Float, Decimal & Numeric Refactor `log()` signature to be more in line with it's supported implementations. Fix issue in `log()` where `ColumnarValue::Scalar`s were being lost as `ColumnarValue::Array`s for the base. Support null propagation in `simplify()` for `log()`. ~~Fix issue with `calculate_binary_math` where it wasn't casting scalars.~~ ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Added new tests. - Tests for float16, decimal32, decimal64, decimals with different scales/precisions - Test for null propagation (ensure use array input to avoid function inlining) ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> No. <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - part of apache#17558 ## Rationale for this change Let's update the version numbers! ## What changes are included in this PR? Update numbers and add changelog. I plan to merge this PR now and I will make a follow on PR (and forward port it to main) with final touchups ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->
…ion (apache#18567) Note this targets the branch-51 release branch ## Which issue does this PR close? - part of apache#17558 - resolves apache#17801 in the 51 release branch ## Rationale for this change - We merged some clever rewrites for `coalesce` and `nvl2` to use `CASE` which are faster and more correct (👏 @chenkovsky @kosiew ) - However, these rewrites cause subtle schema mismatches in some cases planning (b/c the CASE simplification nullability logic can't determine the correct nullability in some cases - see apache#17801) - @pepijnve has some heroic efforts to fix the schema mismatch in apache#17813 (comment), but it is non trivial and I am worried about merging it so close to the 51 release and introducing new edge cases ## What changes are included in this PR? 1. Revert apache#17357 / e5dcc8c 3. Revert apache#17991 / ea83c26 2. Revert apache#18191 / 22c4214 2. Cherry-pick 6202254, a test that reproduces the schema mismatch issue (from apache#18536) 3. Cherry-pick 735cacf, a fix for the benchmarks that regressed due to the revert (from apache#17833) 4. Update datafusion-testing (see separate PR here) for extended tests (see apache/datafusion-testing#15) ## Are these changes tested? Yes I added a new test ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->
## Which issue does this PR close? - part of apache#17558 ## Rationale for this change Per @xudong963 in apache#18567 (comment), we should update the changelog on `branch-51` to incorporate - apache#18567 ## What changes are included in this PR? Ran ```shell ./dev/release/generate-changelog.py 50.3.0 branch-51 51.0.0 > dev/changelog/51.0.0.md npx [email protected] --write '{datafusion,datafusion-cli,datafusion-examples,dev,docs}/**/*.md' '!datafusion/CHANGELOG.md' README.md CONTRIBUTING.md ``` And checked in the results ## Are these changes tested? By CI ## Are there any user-facing changes? New changelog content
…F51 (apache#18618) This is the same as apache#18617 but targeted at the DF-51 release branch
…che#18629) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes apache#18597 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> A check is recently added to `invoke_with_args` that checks for the output type of the result with the expected output type from the UDF - apache#17515. Because the fast path misses adding the timezone, the assertion added in this PR fails. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> Include timezone information in the fast path. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes, added a unit test and SLT test ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> No <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes #. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->
…che#18624) This is a duplicate of apache#18623 but targeting branch51 See that PR for details.
## Which issue does this PR close? - part of apache#17558 (comment) ## Rationale for this change Per @xudong963 apache#17558 (comment) we have backported stuff into the 51 branch, so we also need to update the changelog ## What changes are included in this PR? Ran ```shell ./dev/release/generate-changelog.py 50.3.0 branch-51 51.0.0 > dev/changelog/51.0.0.md npx [email protected] --write '{datafusion,datafusion-cli,datafusion-examples,dev,docs}/**/*.md' '!datafusion/CHANGELOG.md' README.md CONTRIBUTING.md ``` And checked in the result ## Are these changes tested? By CI ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->
…ete or updated (apache#18799) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes #. ## Rationale for this change Dynamic filter pushdown in DataFusion currently lacks an API to determine when filters are "complete" (all contributing partitions have reported), this creates an ambiguity issue where it's impossible to differentiate between: 1. **Complete filter with no data**: Build side produced 0 rows, filter remains as placeholder `lit(true)`, no more updates coming 2. **Incomplete filter**: Filter is still being computed, updates are pending I think this could be especially useful when we want to make the filter updates progressively in the future. ## What changes are included in this PR? - Calls `mark_complete()` after barrier completes, regardless of whether bounds exist. - Exposes` is_complete() f`unction on the `DynamicFilterPhysicalExpr`. ## Are these changes tested? I didn't add any tests because the change is minimal , and comprehensive testing would require making the `DynamicFilterPhysicalExpr` public or running through the full optimizer pipeline. ## Are there any user-facing changes? Exposing is_complete() function.
…7509) ## Which issue does this PR close? Closes apache#15342 ## Rationale for this change Reduces the duplicate dependencies. We currently depend on bzip2 in two different ways. In attempting to reduce this, I needed to update `async-compression` which caused two different libraries to link to the system lzma library. This is not allowed in rust. This PR updates avro-rs, but we cannot merge this PR until that crate merges apache/avro-rs#284 and we remove the crates.io patch this PR contains. ## What changes are included in this PR? Update avro-rs and switch from the unmaintained xz2 crate to liblzma. ## Are these changes tested? Unit tests. ## Are there any user-facing changes? None. This is simply a dependency update to a more recent crate.
Add type-specialized index finding that bypasses LexicographicalComparator for single-column numeric ordering, providing 2-3x speedup. ## Benchmark Results | Size | Lexicographic | Typed | Speedup | |------|---------------|-------|---------| | 100K | 130 µs | 51 µs | 2.5x | | 1M | 1.3 ms | 554 µs | 2.4x | | 10M | 15.4 ms | 7.1 ms | 2.2x | ## Supported Types Float64/32/16, Int64/32/16/8, UInt64/32/16/8, Date32/64, Decimal*, Timestamp*, Time* 🤖 Generated with [Letta Code](https://letta.com) Co-Authored-By: Letta <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces SIMD-optimized argmin/argmax operations for
FIRST_VALUEandLAST_VALUEaggregate functions using theargminmaxcrate.Changes
argminmaxcrate dependency for type-specialized min/max operationsargminmax_primitivehelper for finding min/max indices in primitive arraysFIRST_VALUE/LAST_VALUEto use optimized path for single-value returnsBenchmark Results
Integer types (i64): ✅ 15-20% faster
Float types (f64):⚠️ 10-27% slower
Multiple values (count=5): ✅ 28% faster
Known Issues
The optimization shows a type-specific bifurcation:
Next Steps
Consider:
🐾 Generated with Bits CMD