Support get partition table with filter

[Fokko]

Feature Request / Improvement

Migration of issue apache/iceberg#8619

[puchengy]

Hello @Fokko, here is my use case:

1. Given a table, find out all the partitions it has.
2. Given a partition filter, check if such partition exist in the table.

Thank you!

[Fokko]

@puchengy The problem with Iceberg is that the partition is more of a logical concept, rather than a physical path like in a Hive table. What do you think of passing in a predicate, and letting the Airflow sensor pass if there are rows?

For example, you could go from a daily to an hourly partition. Then you would get:

2023-01-01T00:00:00
2023-01-02T00:00:00
2023-01-03T00:00:00
2023-01-03T23:00:00 # Changed from daily to hourly
2023-01-04T00:00:00
2023-01-04T01:00:00
2023-01-04T02:00:00
2023-01-04T03:00:00

[puchengy]

What do you think of passing in a predicate, and letting the Airflow sensor pass if there are rows?

@Fokko That works. This is actually what we are doing (but for legacy_python) pinterest/iceberg@7d8d65d

Would this be something we can implement in the upstream? Thanks

[puchengy]

@Fokko gentle ping, thanks ^

[Fokko]

@puchengy Yes, certainly. Would this be something that you're interested in working on? From the snapshot, we can load the manifest list, and from there the manifests themselves, which contain the partition information

[puchengy]

@Fokko Yes, I can help. Thanks.

[pp-akursar]

I was looking for something comparable to spark's partitions metadata table, which lets me do something like this

SELECT partition, last_updated_snapshot_id, last_updated_at
FROM prod.db.table.partitions
WHERE partition.foo='bar'

to determine if and when a partition was updated, and came across this issue. It sounds like this could be provided by this Feature Request if it includes a ManifestReader with filtering features like the linked code in legacy_python. Is that correct? If not, I will try to raise as a separate issue.

[sungwy]

Partitions table was added in: #603