Writable Iceberg tables with REST catalog support #68

sfc-gh-okalaci · 2025-11-17T12:55:48Z

In this PR, we add support for writable REST catalog tables. The API may still change, so I refrain documenting it in the PR description. Once we finalise the APIs, the best way would be to put under https://github.com/Snowflake-Labs/pg_lake/blob/main/docs/iceberg-tables.md, so stay tuned for that.

In the earlier set of PRs (#47, #49, #51, #52 and #56) we prototyped adding support for writable rest catalog in different stages. However, it turns out that a single PR is better to tackle these very much related commits. It seemed overkill to maintain these set of PRs.

This new table type shares almost the same architecture with iceberg tables catalog=postgres.

Similarities

Tables are tracked the pg_lake catalogs such as lake_table.files, lake_table.data_file_column_stats and lake_iceberg.tables_internal.
All metadata handling follows ApplyIcebergMetadataChanges() logic. Instead of generating a new metadata.json as we do for catalog=postgres, for these tables we collect the changes happened in the transaction, and apply to the REST catalog right after it is committed in Postgres.

Differences

The metadata_location column in lake_iceberg.tables_internal is always NULL
Does not support RENAME TABLE / SET SCHEMA etc.

Some other notes on the implementation & design:

We first COMMIT in Postgres, then in post-commit hook, send a POST request to REST catalog. So, it is possible that the changes are committed in Postgres, but not in REST catalog. This is a known limitation, and we'll have follow-up PRs to make sure we can recover from this situation.
Creating a table and modifying it in the same Postgres transaction cannot be committed atomically in REST catalog. There is no such API in REST catalog. So, there are some additional error scenarios where table creation committed in REST catalog, say not the full CTAS. This is an unfortunate limitation that we inherit from REST catalog APIs.
Our implementation currently assumes that the Postgres is the single-writer to this table in the REST catalog. So, a concurrent modification breaks the table from Postgres side. For now, this is the current state. We plan to improve it in the future.

TODO:

Especially useful for #68

Both Spark and Polaris follows this. Before this commit, we blindly added partition specs, even if the same spec existed before. Now, we are switching to a more proper way: if the same spec exists in the table earlier, we simply use that instead of adding one more spec. This is especially useful for #68, where Polaris throws an error if we try to send a partition spec that already exists. Here `the same spec` means a spec that has the exact same fields with an existing spec in the table. To give an example, previously we created 3 specs for the following, now 2: ```sql ALTER TABLE t OPTION (ADD partition_by='bucket(10,a)'); .. ALTER TABLE t OPTION (ADD partition_by='bucket(20,a)'); .. -- back to 10, this does not generate a new spec anymore ALTER TABLE t OPTION (ADD partition_by='bucket(10,a)'); ```

Both Spark and Polaris follows this. Before this commit, we blindly added partition specs, even if the same spec existed before. Now, we are switching to a more proper way: if the same spec exists in the table earlier, we simply use that instead of adding one more spec. This is especially useful for #68, where Polaris throws an error if we try to send a partition spec that already exists. Here `the same spec` means a spec that has the exact same fields with an existing spec in the table. To give an example, previously we created 3 specs for the following, now 2: ```sql ALTER TABLE t OPTION (ADD partition_by='bucket(10,a)'); .. ALTER TABLE t OPTION (ADD partition_by='bucket(20,a)'); .. -- back to 10, this does not generate a new spec anymore ALTER TABLE t OPTION (ADD partition_by='bucket(10,a)'); ``` On Polaris, before this commit, we'd get the following, given Polaris thinks this spec already exists: ``` alter table t (set partition_by 'bucket(10,a)'); WARNING: HTTP request failed (HTTP 400) DETAIL: Cannot set last added spec: no spec has been added HINT: The rest catalog returned error type: ValidationException ``` Signed-off-by: Onder KALACI <[email protected]>

Both Spark and Polaris follows this. Before this commit, we blindly added partition specs, even if the same spec existed before. Now, we are switching to a more proper way: if the same spec exists in the table earlier, we simply use that instead of adding one more spec. This is especially useful for #68, where Polaris throws an error if we try to send a partition spec that already exists. Here `the same spec` means a spec that has the exact same fields with an existing spec in the table. To give an example, previously we created 3 specs for the following, now 2: ```sql ALTER TABLE t OPTION (ADD partition_by='bucket(10,a)'); .. ALTER TABLE t OPTION (ADD partition_by='bucket(20,a)'); .. -- back to 10, this does not generate a new spec anymore ALTER TABLE t OPTION (ADD partition_by='bucket(10,a)'); ``` On Polaris, before this commit, we'd get the following, given Polaris thinks this spec already exists: ``` alter table t (set partition_by 'bucket(10,a)'); WARNING: HTTP request failed (HTTP 400) DETAIL: Cannot set last added spec: no spec has been added HINT: The rest catalog returned error type: ValidationException ``` Signed-off-by: Onder KALACI <[email protected]> (cherry picked from commit 75b4eda)

Both Spark and Polaris follows this. Before this commit, we blindly added partition specs, even if the same spec existed before. Now, we are switching to a more proper way: if the same spec exists in the table earlier, we simply use that instead of adding one more spec. This is especially useful for #68, where Polaris throws an error if we try to send a partition spec that already exists. Here `the same spec` means a spec that has the exact same fields with an existing spec in the table. To give an example, previously we created 3 specs for the following, now 2: ```sql ALTER TABLE t OPTION (ADD partition_by='bucket(10,a)'); .. ALTER TABLE t OPTION (ADD partition_by='bucket(20,a)'); .. -- back to 10, this does not generate a new spec anymore ALTER TABLE t OPTION (ADD partition_by='bucket(10,a)'); ``` On Polaris, before this commit, we'd get the following, given Polaris thinks this spec already exists: ``` alter table t (set partition_by 'bucket(10,a)'); WARNING: HTTP request failed (HTTP 400) DETAIL: Cannot set last added spec: no spec has been added HINT: The rest catalog returned error type: ValidationException ``` Signed-off-by: Onder KALACI <[email protected]>

sfc-gh-mslot · 2025-11-28T13:35:01Z

pg_lake_table/src/ddl/alter_table.c

 }

+static bool
+AlterTableAddColumnDefaultForRestTable(Oid relationId, AlterTableCmd *cmd)


should this be something like IsAlterTableAddColumnDefaultForRestTable?

sfc-gh-mslot · 2025-11-28T13:36:18Z

pg_lake_table/src/ddl/alter_table.c

+	if (renameStmt->renameType == OBJECT_SCHEMA)
+	{
+		/* we are in post-process, use new name */
+		ErrorIfAnyRestCatalogTablesInSchema(renameStmt->newname);


should we guard this by extension existing? otherwise we might break postgres

we normally escape from that by checking FDW names and such.

sfc-gh-mslot · 2025-11-28T13:38:21Z

pg_lake_table/src/transaction/track_iceberg_metadata_changes.c

+
+	while ((requestPerTable = hash_seq_search(&status)) != NULL)
+	{
+		/* TODO: can we ever have multiple catalogs? */


not an unreasonable assumption, but one we can ignore for now, we could one day end up wanting tables with custom endpoints. (probably not specifically catalog)

sfc-gh-mslot · 2025-11-28T13:39:24Z

pg_lake_table/src/transaction/track_iceberg_metadata_changes.c

+}
+
+void
+PostAllRestCatalogRequests(void)


probably good to add some comments here, especially that this function should never error

sfc-gh-mslot

Overall looks pretty good.

I like being able to do things like create table test_rest using iceberg with (catalog = 'rest') as select 5; and then query using other tools.

sfc-gh-mslot · 2025-11-28T13:51:28Z

pg_lake_iceberg/src/iceberg/catalog.c

 {
 	bool		forUpdate = false;
-	char	   *columnName = "metadata_location";
+	char	   *columnName = "table_name";


probably variable name below should also change then

sfc-gh-mslot · 2025-11-28T13:54:31Z

pg_lake_table/src/transaction/track_iceberg_metadata_changes.c

+		requestPerTable->relationId = relationId;
+
+		requestPerTable->catalogName = MemoryContextStrdup(TopTransactionContext, GetRestCatalogName(relationId));
+		requestPerTable->catalogNamespace = MemoryContextStrdup(TopTransactionContext, GetRestCatalogNamespace(relationId));


some minor OOM hazards here in subtransactions maybe, might be good to add an isValid flag.

This was referenced Nov 17, 2025

Create table in REST catalog #47

Closed

Modify table in REST catalog #49

Closed

Schema changes in REST catalog #51

Closed

Partition changes in REST catalog #52

Closed

Remove Snapshots in REST catalog #56

Closed

sfc-gh-okalaci marked this pull request as draft November 17, 2025 13:03

sfc-gh-okalaci force-pushed the onder/register_namespace_on_rest_table_create branch from 02fcb9a to 0f1c010 Compare November 18, 2025 14:56

sfc-gh-okalaci force-pushed the onder/rest_catalog_end_to_end branch from 64203e0 to 7f200e8 Compare November 18, 2025 15:17

sfc-gh-okalaci mentioned this pull request Nov 19, 2025

Support data file pruning for external iceberg tables #70

Open

1 task

sfc-gh-okalaci added a commit that referenced this pull request Nov 19, 2025

Add a GUC to log HTTP requests

3515a96

Especially useful for #68

sfc-gh-okalaci mentioned this pull request Nov 19, 2025

Add a GUC to log HTTP traffic #71

Merged

sfc-gh-okalaci added a commit that referenced this pull request Nov 19, 2025

Add a GUC to log HTTP requests

b5404f6

Especially useful for #68

sfc-gh-okalaci force-pushed the onder/rest_catalog_end_to_end branch from ac8f08e to 9e7183a Compare November 19, 2025 16:54

sfc-gh-okalaci force-pushed the onder/register_namespace_on_rest_table_create branch from 0f1c010 to 2978252 Compare November 19, 2025 17:37

sfc-gh-okalaci added a commit that referenced this pull request Nov 19, 2025

Add a GUC to log HTTP requests

cd5c5f0

Especially useful for #68

sfc-gh-okalaci force-pushed the onder/rest_catalog_end_to_end branch from 9e7183a to eed829c Compare November 19, 2025 17:38

Base automatically changed from onder/register_namespace_on_rest_table_create to main November 20, 2025 05:53

sfc-gh-okalaci added a commit that referenced this pull request Nov 20, 2025

Add a GUC to log HTTP requests

2a9c1a9

Especially useful for #68

sfc-gh-okalaci added a commit that referenced this pull request Nov 20, 2025

Add a GUC to log HTTP requests

6b203cf

Especially useful for #68

sfc-gh-okalaci force-pushed the onder/rest_catalog_end_to_end branch from eed829c to 6c9cee7 Compare November 20, 2025 07:10

sfc-gh-okalaci added a commit that referenced this pull request Nov 20, 2025

Add a GUC to log HTTP requests

7dae279

Especially useful for #68

sfc-gh-okalaci added a commit that referenced this pull request Nov 20, 2025

Add a GUC to log HTTP requests (#71)

697be05

Especially useful for #68

sfc-gh-okalaci force-pushed the onder/rest_catalog_end_to_end branch from 6c9cee7 to 1e80069 Compare November 21, 2025 11:32

sfc-gh-okalaci mentioned this pull request Nov 21, 2025

Do not add a partition spec if it already exists #79

Merged

sfc-gh-okalaci added 11 commits November 26, 2025 09:20

Minor changes

9ef149c

WIP, drop table

acf55c2

drop table in rest catalog

edf14fa

Move writable tests to its own file

c2310de

Finish asserts

15f86ad

Apply required changes for partitioning

a2e8c18

Apply required changes for partitioning

29c57b9

add vacuum test

a3ff0a2

DROP TABLE tests

fefa946

Transactions involving different table types

30c0cde

rebase on top 70

3cf0d5e

sfc-gh-okalaci force-pushed the onder/rest_catalog_end_to_end branch from 1d626ca to 3cf0d5e Compare November 26, 2025 07:14

sfc-gh-okalaci added 9 commits November 26, 2025 11:28

Some more serious modification&vacuum tests

bcae7f9

vacuum (iceberg)

8be8d14

Add tests for types

f9d98a9

Dont run in installcheck

0ed92c6

DDL testts & errors, part-1

c123713

add missing installcheck

d7fb557

fix typo

e41fc5c

Finish DDLs

e78a2a7

autovacuum test

3bddb59

sfc-gh-okalaci marked this pull request as ready for review November 28, 2025 05:45

sfc-gh-mslot mentioned this pull request Nov 28, 2025

Improve external Iceberg compatibility #83

Open

sfc-gh-mslot reviewed Nov 28, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Writable Iceberg tables with REST catalog support #68

Writable Iceberg tables with REST catalog support #68

sfc-gh-okalaci commented Nov 17, 2025 •

edited

Loading

Uh oh!

sfc-gh-mslot Nov 28, 2025

Uh oh!

sfc-gh-mslot Nov 28, 2025

Uh oh!

sfc-gh-mslot Nov 28, 2025 •

edited

Loading

Uh oh!

sfc-gh-mslot Nov 28, 2025

Uh oh!

sfc-gh-mslot left a comment

Uh oh!

sfc-gh-mslot Nov 28, 2025

Uh oh!

sfc-gh-mslot Nov 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Writable Iceberg tables with REST catalog support #68

Are you sure you want to change the base?

Writable Iceberg tables with REST catalog support #68

Conversation

sfc-gh-okalaci commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Similarities

Differences

Some other notes on the implementation & design:

TODO:

Uh oh!

sfc-gh-mslot Nov 28, 2025

Choose a reason for hiding this comment

Uh oh!

sfc-gh-mslot Nov 28, 2025

Choose a reason for hiding this comment

Uh oh!

sfc-gh-mslot Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfc-gh-mslot Nov 28, 2025

Choose a reason for hiding this comment

Uh oh!

sfc-gh-mslot left a comment

Choose a reason for hiding this comment

Uh oh!

sfc-gh-mslot Nov 28, 2025

Choose a reason for hiding this comment

Uh oh!

sfc-gh-mslot Nov 28, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sfc-gh-okalaci commented Nov 17, 2025 •

edited

Loading

sfc-gh-mslot Nov 28, 2025 •

edited

Loading