Grouped top k operator #993

kevin-dp · 2025-12-09T14:21:47Z

Followup on #738.
Implements the same strategy as in that PR to support grouped topK but as a new grouped topK operator. The logic regarding an individual topK is extracted into a common TopKState class that is reused by both the original topK operator and the grouped topK operator.

cursor · 2025-12-09T14:21:48Z

Cursor Agent can help with this pull request. Just @cursor in comments and I'll start working on changes in this branch.
_{Learn more about Cursor Agents}

changeset-bot · 2025-12-09T14:21:51Z

🦋 Changeset detected

Latest commit: 190a02b

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 13 packages

Name	Type
@tanstack/db-ivm	Patch
@tanstack/db	Patch
@tanstack/angular-db	Patch
@tanstack/electric-db-collection	Patch
@tanstack/offline-transactions	Patch
@tanstack/powersync-db-collection	Patch
@tanstack/query-db-collection	Patch
@tanstack/react-db	Patch
@tanstack/rxdb-db-collection	Patch
@tanstack/solid-db	Patch
@tanstack/svelte-db	Patch
@tanstack/trailbase-db-collection	Patch
@tanstack/vue-db	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

pkg-pr-new · 2025-12-09T14:23:33Z

More templates

@tanstack/angular-db

npm i https://pkg.pr.new/@tanstack/angular-db@993

@tanstack/db

npm i https://pkg.pr.new/@tanstack/db@993

@tanstack/db-ivm

npm i https://pkg.pr.new/@tanstack/db-ivm@993

@tanstack/electric-db-collection

npm i https://pkg.pr.new/@tanstack/electric-db-collection@993

@tanstack/offline-transactions

npm i https://pkg.pr.new/@tanstack/offline-transactions@993

@tanstack/powersync-db-collection

npm i https://pkg.pr.new/@tanstack/powersync-db-collection@993

@tanstack/query-db-collection

npm i https://pkg.pr.new/@tanstack/query-db-collection@993

@tanstack/react-db

npm i https://pkg.pr.new/@tanstack/react-db@993

@tanstack/rxdb-db-collection

npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@993

@tanstack/solid-db

npm i https://pkg.pr.new/@tanstack/solid-db@993

@tanstack/svelte-db

npm i https://pkg.pr.new/@tanstack/svelte-db@993

@tanstack/trailbase-db-collection

npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@993

@tanstack/vue-db

npm i https://pkg.pr.new/@tanstack/vue-db@993

commit: 190a02b

github-actions · 2025-12-09T14:26:42Z

Size Change: 0 B

Total Size: 88.2 kB

ℹ️ View Unchanged

Filename	Size
`./packages/db/dist/esm/collection/change-events.js`	1.39 kB
`./packages/db/dist/esm/collection/changes.js`	977 B
`./packages/db/dist/esm/collection/events.js`	388 B
`./packages/db/dist/esm/collection/index.js`	3.24 kB
`./packages/db/dist/esm/collection/indexes.js`	1.1 kB
`./packages/db/dist/esm/collection/lifecycle.js`	1.67 kB
`./packages/db/dist/esm/collection/mutations.js`	2.31 kB
`./packages/db/dist/esm/collection/state.js`	3.42 kB
`./packages/db/dist/esm/collection/subscription.js`	2.71 kB
`./packages/db/dist/esm/collection/sync.js`	2.37 kB
`./packages/db/dist/esm/deferred.js`	207 B
`./packages/db/dist/esm/errors.js`	4.19 kB
`./packages/db/dist/esm/event-emitter.js`	748 B
`./packages/db/dist/esm/index.js`	2.66 kB
`./packages/db/dist/esm/indexes/auto-index.js`	742 B
`./packages/db/dist/esm/indexes/base-index.js`	766 B
`./packages/db/dist/esm/indexes/btree-index.js`	1.93 kB
`./packages/db/dist/esm/indexes/lazy-index.js`	1.1 kB
`./packages/db/dist/esm/indexes/reverse-index.js`	513 B
`./packages/db/dist/esm/local-only.js`	837 B
`./packages/db/dist/esm/local-storage.js`	2.1 kB
`./packages/db/dist/esm/optimistic-action.js`	359 B
`./packages/db/dist/esm/paced-mutations.js`	496 B
`./packages/db/dist/esm/proxy.js`	3.75 kB
`./packages/db/dist/esm/query/builder/functions.js`	733 B
`./packages/db/dist/esm/query/builder/index.js`	3.96 kB
`./packages/db/dist/esm/query/builder/ref-proxy.js`	917 B
`./packages/db/dist/esm/query/compiler/evaluators.js`	1.35 kB
`./packages/db/dist/esm/query/compiler/expressions.js`	430 B
`./packages/db/dist/esm/query/compiler/group-by.js`	1.8 kB
`./packages/db/dist/esm/query/compiler/index.js`	1.96 kB
`./packages/db/dist/esm/query/compiler/joins.js`	2 kB
`./packages/db/dist/esm/query/compiler/order-by.js`	1.46 kB
`./packages/db/dist/esm/query/compiler/select.js`	1.07 kB
`./packages/db/dist/esm/query/expression-helpers.js`	1.43 kB
`./packages/db/dist/esm/query/ir.js`	673 B
`./packages/db/dist/esm/query/live-query-collection.js`	360 B
`./packages/db/dist/esm/query/live/collection-config-builder.js`	5.33 kB
`./packages/db/dist/esm/query/live/collection-registry.js`	264 B
`./packages/db/dist/esm/query/live/collection-subscriber.js`	1.74 kB
`./packages/db/dist/esm/query/live/internal.js`	130 B
`./packages/db/dist/esm/query/optimizer.js`	2.56 kB
`./packages/db/dist/esm/query/predicate-utils.js`	2.91 kB
`./packages/db/dist/esm/query/subset-dedupe.js`	921 B
`./packages/db/dist/esm/scheduler.js`	1.3 kB
`./packages/db/dist/esm/SortedMap.js`	1.3 kB
`./packages/db/dist/esm/strategies/debounceStrategy.js`	247 B
`./packages/db/dist/esm/strategies/queueStrategy.js`	428 B
`./packages/db/dist/esm/strategies/throttleStrategy.js`	246 B
`./packages/db/dist/esm/transactions.js`	2.9 kB
`./packages/db/dist/esm/utils.js`	881 B
`./packages/db/dist/esm/utils/browser-polyfills.js`	304 B
`./packages/db/dist/esm/utils/btree.js`	5.61 kB
`./packages/db/dist/esm/utils/comparison.js`	852 B
`./packages/db/dist/esm/utils/cursor.js`	457 B
`./packages/db/dist/esm/utils/index-optimization.js`	1.51 kB
`./packages/db/dist/esm/utils/type-guards.js`	157 B

_{compressed-size-action::db-package-size}

github-actions · 2025-12-09T14:27:18Z

Size Change: 0 B

Total Size: 3.35 kB

ℹ️ View Unchanged

Filename	Size
`./packages/react-db/dist/esm/index.js`	225 B
`./packages/react-db/dist/esm/useLiveInfiniteQuery.js`	1.17 kB
`./packages/react-db/dist/esm/useLiveQuery.js`	1.12 kB
`./packages/react-db/dist/esm/useLiveSuspenseQuery.js`	431 B
`./packages/react-db/dist/esm/usePacedMutations.js`	401 B

_{compressed-size-action::react-db-package-size}

This commit introduces a new operator that allows for topK operations to be performed independently on groups of data. Co-authored-by: kevin.de.porre <[email protected]>

Introduce TopKState<K, T> to encapsulate the common state and operations for managing a single topK window (multiplicity tracking and topK data structure). Both TopKWithFractionalIndexOperator (single instance) and GroupedTopKWithFractionalIndexOperator (one instance per group) now use this helper class, eliminating code duplication. Also extract handleMoveIn and handleMoveOut as standalone functions that can be reused by both operators.

KyleAMathews

Code Review: PR #993

Overall Assessment: Approve ✅

This is excellent foundational work that both refactors existing code AND adds new capability. Like Brigham Young organizing the Saints into companies with captains of tens and hundreds, this PR introduces a hierarchical grouping structure for topK operations while maintaining clean separation of concerns.

Summary

The PR introduces:

TopKArray - Extracted TopK data structure with fractional indexing
TopKState - Encapsulated state management (multiplicity tracking + TopK)
GroupedTopKWithFractionalIndexOperator - Per-group topK windows
Refactored original topKWithFractionalIndex to use shared abstractions

Strengths

Clean Extraction & Reuse: The TopKArray and TopKState classes are cleanly extracted and reusable.
Memory Management: Empty groups are cleaned up to prevent memory leaks via #cleanupGroupIfEmpty.
Extensible Design: The createTopK method is protected and can be overridden for different implementations (e.g., B+ tree).
Stable Ordering: The createKeyedComparator provides deterministic ordering by using the row key as a tie-breaker.
Comprehensive Test Suite: Tests cover all major scenarios including incremental updates, removals, multiple groups, offset support, and window movement.

Minor Suggestions

topKArray.ts:183 - The TODO comment about refactoring insert/delete is a good observation. The window-shifting logic could potentially be unified.
topKState.ts:61-65 - The error message for B+-tree move could be more informative about what the caller should do instead.
groupedTopKWithFractionalIndex.ts:107 - The TODO comment should be resolved or turned into an issue.

Question

For the grouped operator, when all items in a group are removed and then items are added back to that same group, does the fractional indexing start fresh? This seems correct (new group = new TopKArray = fresh fractional indices), but worth confirming.

Solid, well-organized code that enables important hierarchical data projection capabilities! 🏛️

cursoragent and others added 4 commits December 10, 2025 10:20

feat: Add groupedTopKWithFractionalIndex operator

e0d072e

This commit introduces a new operator that allows for topK operations to be performed independently on groups of data. Co-authored-by: kevin.de.porre <[email protected]>

ci: apply automated fixes

506cc8d

ci: apply automated fixes

6193beb

kevin-dp force-pushed the cursor/grouped-top-k-operator-34bd branch from fc2fc60 to 1b9d7ea Compare December 10, 2025 09:24

Refactoring of the topK operator and grouped topK operator

88dbcf6

kevin-dp force-pushed the cursor/grouped-top-k-operator-34bd branch from 1b9d7ea to 88dbcf6 Compare December 10, 2025 09:29

autofix-ci bot and others added 3 commits December 10, 2025 09:43

ci: apply automated fixes

fef17b6

Improve unit tests

f4e1cd6

ci: apply automated fixes

6e03bd8

kevin-dp mentioned this pull request Dec 10, 2025

Add grouped order by operator #997

Open

kevin-dp marked this pull request as ready for review December 10, 2025 12:54

kevin-dp requested a review from samwillis December 10, 2025 12:54

Changeset

190a02b

kevin-dp moved this to Ready for review in TanStack DB 1.0.0 release Jan 6, 2026

kevin-dp added this to TanStack DB 1.0.0 release Jan 6, 2026

KyleAMathews reviewed Jan 8, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Grouped top k operator #993

Grouped top k operator #993

kevin-dp commented Dec 9, 2025 •

edited

Loading

Uh oh!

cursor bot commented Dec 9, 2025

Uh oh!

changeset-bot bot commented Dec 9, 2025 •

edited

Loading

Uh oh!

pkg-pr-new bot commented Dec 9, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Dec 9, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Dec 9, 2025 •

edited

Loading

Uh oh!

KyleAMathews left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Grouped top k operator #993

Are you sure you want to change the base?

Grouped top k operator #993

Conversation

kevin-dp commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor bot commented Dec 9, 2025

Uh oh!

changeset-bot bot commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

pkg-pr-new bot commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KyleAMathews left a comment

Choose a reason for hiding this comment

Code Review: PR #993

Summary

Strengths

Minor Suggestions

Question

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kevin-dp commented Dec 9, 2025 •

edited

Loading

changeset-bot bot commented Dec 9, 2025 •

edited

Loading

pkg-pr-new bot commented Dec 9, 2025 •

edited

Loading

github-actions bot commented Dec 9, 2025 •

edited

Loading

github-actions bot commented Dec 9, 2025 •

edited

Loading