Use partition information in manifest file to perform filter pushdown by Tmonster · Pull Request #602 · duckdb/duckdb-iceberg

Tmonster · 2025-11-24T10:07:37Z

Given a table with many small files and many snapshots, it's possible some kind of compaction is performed. During this time, new manifest lists and manifest files (and potentially parquet files) are written. Somewhere in the Spark Rest catalog, new manifest files that are written during compaction do not contain upper and lower bound values for certain columns. What spark will do, however, is write partition information in the partitions field of the manifest columns. DuckDB only checks upper and lower bound values of columns in manifest files. Because of this, DuckDB will end up reading waaaaay too many files when there is a filter on a partitioned column.

To fix this problem, DuckDB will now read partition information in manifest lists.

Some other notes.

I could not manage a way to consistently reproduce this issue, so I had to add a persistent data. I have stepped through the code and tested that this works
I also added a test where a table has multiple partitions, that caused a bug in Iceberg to DuckLake, which I have now also fixed.
Also added some helper scripts to make it easier to create a persistent table from one in an IRC catalog somewhere

… types (i.e one filter on partition column another not, OR vs. AND)

Tishj · 2025-11-24T20:00:40Z

 		for (auto &field : partition.fields) {
 			columns.push_back(DuckLakePartitionColumn(field));
 		}
+		partition_id = partition.spec_id;


I don't think this is correct?
See the comment:
//! The id is assigned after we've processed all tables

This already happens later, see FinalizeEntry

Tishj · 2025-11-24T20:03:15Z

Thanks, I really wasn't sure what purpose the partition of the data file served, they feel redundant when we already have stats on the manifest file. But I guess the partition can change and through the partition we preserve the partition values that were active when the data was added?

Tishj · 2025-11-24T20:06:40Z

+				auto stats = IcebergPredicateStats();
+				bool found_parition_field = false;
+				for (auto &partition_val : file.partition_values) {
+					auto source_id = partition_val.first;


This shouldn't be called source_id, the partition values are matched on the field_id of the partition:

Partition data tuple, schema based on the partition spec output using partition field ids for the struct field ids

Tishj · 2025-11-24T20:07:21Z

+
+				// initialize dummy stats
+				auto stats = IcebergPredicateStats();
+				bool found_parition_field = false;


found_partition_field

Tishj · 2025-11-24T20:12:28Z

+				}
+
+				if (!found_parition_field) {
+					break;


I'm not sure if breaking or continuing is the best course of action here, can't we still determine if a file should be skipped based on the other partitions that are present?

Tishj · 2025-11-24T20:13:17Z

+					break;
+				}
+
+				auto result_type = field.transform.GetSerializedType(column.type);


This is unused?

I went looking to see if the partition values adhere to the same schema evolution rules as lower/upper bounds of a data file, but they do not:

Type promotion is not allowed for a field that is referenced by source-id or source-ids of a partition field if the partition transform would produce a different value after promoting the type

So this is correct 👍

I went looking at iceberg-python just to check, but this is so abstract..
https://github.com/apache/iceberg-python/blob/main/pyiceberg/table/__init__.py#L1929
I can't make out what they're actually doing 🙃

But then this:

def _build_partition_evaluator(self, spec_id: int) -> Callable[[DataFile], bool]: spec = self.table_metadata.specs()[spec_id] partition_type = spec.partition_type(self.table_metadata.schema()) partition_schema = Schema(*partition_type.fields) partition_expr = self.partition_filters[spec_id] # The lambda created here is run in multiple threads. # So we avoid creating _EvaluatorExpression methods bound to a single # shared instance across multiple threads. return lambda data_file: expression_evaluator(partition_schema, partition_expr, self.case_sensitive)(data_file.partition)

Makes me think that schema evolution is applied, and we need to potentially up-cast the stored partition value to the current type of the partition spec's field ..

Type promotion is not allowed for a field that is referenced by source-id or source-ids of a partition field if the partition transform would produce a different value after promoting the type

I think I get what is happening here.
If you promote an int column that is partitioned by bucket(a) to string, then the new partition values on strings are different than on ints, (i.e 24 vs "24"). When querying with a filter on the string partitioned column (where a = "24"), DuckDB will assume the constant value is a string, and not an int, and the transform for partition pruning may be incorrect.

For safe type promotions (ie int -> long) existing data files remain valid. To figure out what kind of Transform to apply to the filter constant, you can look at the partition_id in each entry of a manifest_list. Every manifest file has only data files from one partition. Using that partition_id, we can find out what transform was used to write the data files, and apply that to the expression constant to see if we can filter out certain data files/manifest files.

The spec also says that a manifest file can only list data files from the same partition spec. https://iceberg.apache.org/spec/#partitioning

Don't know if this helps at all

Yea you're right, so schema evolution can happen, as long as it doesn't change the meaning of the partition transform

Then I think this logic needs to up-cast the stored partition Value, don't you think?

So just to provide an update for future reference:
I don't think any upcasting is necessary

The lower/upper bounds in the data_file are serialized according to the type of the column at the time of file creation.
To properly deserialize them we need to infer what this type was, in case schema evolution took place. (i.e INTEGER -> BIGINT, the bounds are inferred to be INTEGER when the size of the blob is 4 (sizeof(int32_t))

Since the partition values have type information stored, there is no need for any inference.

Tmonster · 2025-11-25T18:27:34Z

@Tishj Also added some more logic around null partitions, I can also take it out and put it in a separate PR if needed

Tishj

Thanks, this looks great, the addition of null handling is great as well 👍

Tmonster · 2025-11-26T11:37:57Z

Thanks, I'm gonna add some more tests though around promoting partitioned types just to make sure.

I.e
a int -> a varchar (partitioned on identity transform)
a int -> a varchar (partitioned on truncate transform)
a int -> a varchar (partitioned on void transform)

identity, truncate, and void are the only transforms where the resulting type can change
We can see what happens to upper and lower bound values, along with the partition values.

Tmonster · 2025-11-26T16:31:10Z

So int -> varchar type promotion is not supported in Iceberg, so we don't have to worry about that. (source)

But I will still add a case where a data file cannot be filtered out based on the partition, but it can be filtered out based on upper and lower bounds.

…rtition matches but not the upper/lower bounds

Tmonster added 10 commits November 18, 2025 16:08

use partition information of data files when pruning

9dff630

set stats.has_not_null

846f57a

add data for test

8af2d15

added test to test evolved partitions

b815081

remove unnecessary data file. Check more pruning and different filter…

2c08a3e

… types (i.e one filter on partition column another not, OR vs. AND)

remove last TODOs

cb91c38

move helper file

581648e

remove unnecessary code again.

a0a1d43

need to use unique ptrs to ducklake partition info

50beb13

remove some unneeded files and add more comments

5857846

Tmonster requested a review from Tishj November 24, 2025 10:07

Tishj reviewed Nov 24, 2025

View reviewed changes

Tmonster added 5 commits November 25, 2025 13:40

PR comments, also consider null partitions

5eb16a0

some better logic around upper and lower bounds

3a5ca37

more tests for null filter pushdown

6f60370

fix up test file

2fbe976

fix other tests

d0975eb

Tmonster requested a review from Tishj November 25, 2025 18:27

Tishj approved these changes Nov 26, 2025

View reviewed changes

Tmonster added 3 commits November 27, 2025 11:32

more tests around filter pushdown, but now filter pushdown where a pa…

b987b0e

…rtition matches but not the upper/lower bounds

add update to nessie script

7d34392

sudo not suto

0573312

Tmonster merged commit 79d34f4 into duckdb:v1.4-andium Nov 28, 2025
19 checks passed

Tmonster deleted the use_partition_information_in_manifest_file branch November 28, 2025 09:19

Conversation

Tmonster commented Nov 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tishj commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tishj Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tishj Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tmonster commented Nov 25, 2025

Uh oh!

Tishj left a comment

Choose a reason for hiding this comment

Uh oh!

Tmonster commented Nov 26, 2025

Uh oh!

Tmonster commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Tishj commented Nov 24, 2025 •

edited

Loading

Tishj Nov 24, 2025 •

edited

Loading

Tishj Nov 26, 2025 •

edited

Loading

Tmonster commented Nov 26, 2025 •

edited

Loading