Skip to content

Support data file pruning for external iceberg tables #70

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: aykut/read-only-refactor
Choose a base branch
from
Open

Support data file pruning for external iceberg tables #70

wants to merge 1 commit into from

Conversation

@sfc-gh-abozkurt
Copy link
Collaborator

@sfc-gh-abozkurt sfc-gh-abozkurt commented Nov 18, 2025 •
edited
Loading

  • Add tests.

Closes #55.

sfc-gh-abozkurt
sfc-gh-abozkurt commented Nov 18, 2025
View reviewed changes
pg_lake_engine/include/pg_lake/util/rel_utils.h Outdated
#define PG_LAKE_ICEBERG_SERVER_NAME "pg_lake_iceberg"

extern PGDLLEXPORT bool IsAnyLakeForeignTableById(Oid foreignTableId);
extern PGDLLEXPORT char *GetForeignTablePath(Oid foreignTableId);
Copy link
Collaborator Author

@sfc-gh-abozkurt sfc-gh-abozkurt Nov 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved from other location

sfc-gh-okalaci
sfc-gh-okalaci reviewed Nov 19, 2025
View reviewed changes
Copy link
Collaborator

@sfc-gh-okalaci sfc-gh-okalaci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, interesting and useful PR!

I think we should be able to split this PR into two:

  • Refactors, new utility functions, no logic change PR
  • The PR that adds the external partition pruning

The first one can be useful to have before we ship #68 such that the flow follows that from start. We can discuss the details of the APIs in that PR?

@sfc-gh-abozkurt sfc-gh-abozkurt mentioned this pull request Nov 20, 2025
Merged
@sfc-gh-abozkurt sfc-gh-abozkurt force-pushed the aykut/prune-external-iceberg branch from a140bb6 to 0b4bbca Compare November 20, 2025 12:02
@sfc-gh-abozkurt sfc-gh-abozkurt changed the base branch from main to aykut/refactor-internal-external November 20, 2025 12:03
sfc-gh-abozkurt
sfc-gh-abozkurt commented Nov 20, 2025
View reviewed changes
pg_lake_table/src/fdw/partition_transform.c
{
DataFileSchema *schema = GetDataFileSchemaForTable(relationId);

transform->sourceField = GetDataFileSchemaFieldById(schema, specField->source_id);
Copy link
Collaborator Author

@sfc-gh-abozkurt sfc-gh-abozkurt Nov 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can source field exist at an old schema?

Copy link
Collaborator Author

@sfc-gh-abozkurt sfc-gh-abozkurt Nov 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We disable dropping columns if they exist in any partition spec like below:

postgres=# create table test(a int) using iceberg;                                                                                                                 
CREATE TABLE
postgres=# ALTER TABLE test OPTIONS (add partition_by 'bucket(2,a)');
ALTER TABLE
postgres=# alter table test add column b int;
ALTER TABLE
postgres=# alter table test drop column a;
ERROR:  cannot drop column "a" from table test because it is used in the partition spec

We cant handle such flow for external tables with this PR.

Copy link
Collaborator Author

@sfc-gh-abozkurt sfc-gh-abozkurt Nov 20, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should skip such partition spec fields anyway. Those do not exist in the current postgres schema and pruning wont apply them anyway.

sfc-gh-abozkurt
sfc-gh-abozkurt commented Nov 20, 2025
View reviewed changes
pg_lake_table/src/fdw/schema_operations/field_id_mapping_catalog.c

DataFileSchemaField *schemaField = GetDataFileSchemaFieldById(schema, fieldId);

char *attrName = pstrdup(schemaField->name);
Copy link
Collaborator Author

@sfc-gh-abozkurt sfc-gh-abozkurt Nov 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

attribute name is safe for read only external tables

@sfc-gh-abozkurt sfc-gh-abozkurt force-pushed the aykut/refactor-internal-external branch 5 times, most recently from a33ede2 to 407d6b6 Compare November 20, 2025 14:51
@sfc-gh-abozkurt sfc-gh-abozkurt force-pushed the aykut/prune-external-iceberg branch 5 times, most recently from 4226886 to f01087b Compare November 20, 2025 16:53
@sfc-gh-abozkurt sfc-gh-abozkurt force-pushed the aykut/refactor-internal-external branch from 407d6b6 to 90f7ecc Compare November 21, 2025 03:07
@sfc-gh-abozkurt sfc-gh-abozkurt force-pushed the aykut/prune-external-iceberg branch from f01087b to 014a78f Compare November 21, 2025 03:12
@sfc-gh-abozkurt sfc-gh-abozkurt force-pushed the aykut/refactor-internal-external branch from 90f7ecc to 8b40701 Compare November 21, 2025 09:48
@sfc-gh-abozkurt sfc-gh-abozkurt force-pushed the aykut/prune-external-iceberg branch 2 times, most recently from 3721417 to 375ef82 Compare November 21, 2025 10:39
@sfc-gh-abozkurt sfc-gh-abozkurt force-pushed the aykut/refactor-internal-external branch from 8b40701 to 22c7548 Compare November 21, 2025 16:09
@sfc-gh-abozkurt sfc-gh-abozkurt force-pushed the aykut/prune-external-iceberg branch from 375ef82 to 2973ec2 Compare November 21, 2025 16:10
@sfc-gh-abozkurt sfc-gh-abozkurt force-pushed the aykut/refactor-internal-external branch from 22c7548 to 8b9897a Compare November 22, 2025 16:59
@sfc-gh-abozkurt sfc-gh-abozkurt force-pushed the aykut/prune-external-iceberg branch 2 times, most recently from de3f2a5 to 35b125c Compare November 23, 2025 00:20
@sfc-gh-abozkurt sfc-gh-abozkurt marked this pull request as ready for review November 23, 2025 00:20
sfc-gh-abozkurt
sfc-gh-abozkurt commented Nov 23, 2025
View reviewed changes
pg_lake_table/tests/pytests/test_data_file_pruning.py
CREATE TABLE test_data_file_pruning.{table_name}_object_store_read_only ()
USING iceberg WITH (catalog='object_store', read_only=True, catalog_table_name='{table_name}_object_store');
-- CREATE TABLE test_data_file_pruning.{table_name}_rest_read_only ()
Copy link
Collaborator Author

@sfc-gh-abozkurt sfc-gh-abozkurt Nov 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we do not push into rest catalog yet, this will work after we support it.

sfc-gh-abozkurt
sfc-gh-abozkurt commented Nov 23, 2025
View reviewed changes
pg_lake_table/tests/pytests/test_partition_pruning.py
CREATE TABLE test_data_file_pruning.{table_name}_object_store_read_only ()
USING iceberg WITH (catalog='object_store', read_only=True, catalog_table_name='{table_name}_object_store');
-- CREATE TABLE test_data_file_pruning.{table_name}_rest_read_only ()
Copy link
Collaborator Author

@sfc-gh-abozkurt sfc-gh-abozkurt Nov 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we do not push into rest catalog yet, this will work after we support it.

@sfc-gh-abozkurt sfc-gh-abozkurt force-pushed the aykut/prune-external-iceberg branch from 35b125c to 295affc Compare November 24, 2025 12:45
@sfc-gh-abozkurt sfc-gh-abozkurt force-pushed the aykut/refactor-internal-external branch 2 times, most recently from 1b05eb3 to e3858c7 Compare November 24, 2025 20:37
@sfc-gh-abozkurt sfc-gh-abozkurt force-pushed the aykut/prune-external-iceberg branch from 295affc to dfb54fa Compare November 24, 2025 20:53
@sfc-gh-abozkurt sfc-gh-abozkurt changed the base branch from aykut/refactor-internal-external to aykut/read-only-refactor November 24, 2025 20:53
@sfc-gh-abozkurt sfc-gh-abozkurt force-pushed the aykut/read-only-refactor branch from 563b36c to 1e5f0b7 Compare November 25, 2025 20:38
@sfc-gh-abozkurt sfc-gh-abozkurt force-pushed the aykut/prune-external-iceberg branch from dfb54fa to 425c7a8 Compare November 25, 2025 20:39
@sfc-gh-abozkurt sfc-gh-abozkurt force-pushed the aykut/read-only-refactor branch from 1e5f0b7 to aa9de74 Compare November 25, 2025 20:52
@sfc-gh-abozkurt sfc-gh-abozkurt force-pushed the aykut/prune-external-iceberg branch from 425c7a8 to 69a2265 Compare November 25, 2025 20:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sfc-gh-okalaci sfc-gh-okalaci sfc-gh-okalaci left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Support partition pruning for external tables

3 participants

@sfc-gh-abozkurt @sfc-gh-okalaci