Optimize creating indexes when writing partitioned table #5292
Labels
2023_unscheduled
feature request
New feature or request
NoDocumentationNeeded
NoReleaseNotesNeeded
No release notes are needed.
parquet
Related to the Parquet integration
Milestone
In the current code when writing a source table in a key-value partitioned fashion, if user provides indexing columns, we compute the constituent tables to write and then add an index on each of the provided column for each constituent table.
This can be optimized by intersecting with any existing indexes on the original table. We can perform a transform to intersect the index tables' row sets with the constituent row sets, then filter the empty row sets.
Pseudocode for intersecting (courtesy of @rcaudy), probably with bad parentheses, etc:
(Found during #5105)
The text was updated successfully, but these errors were encountered: