Iceberg REST catalog does not delete Puffin stats files when purging the table

[findepi]

Repro steps using QueryRunner as created by io.trino.plugin.iceberg.catalog.rest.TestIcebergTrinoRestCatalogConnectorTest#createQueryRunner

trino:tpch> CREATE TABLE t AS SELECT 1 a;
CREATE TABLE: 1 row

Query 20230316_102953_00021_w7bqc, FINISHED, 2 nodes
http://localhost:8080/ui/query.html?20230316_102953_00021_w7bqc
Splits: 20 total, 20 done (100.00%)
CPU Time: 0.0s total,     0 rows/s,     0B/s, 34% active
Per Node: 0.0 parallelism,     0 rows/s,     0B/s
Parallelism: 0.1
Peak Memory: 2.38KB
0.17 [0 rows, 0B] [0 rows/s, 0B/s]

trino:tpch> ANALYZE t;
ANALYZE
 rows
------
 NULL
(1 row)

Query 20230316_102958_00022_w7bqc, FINISHED, 3 nodes
http://localhost:8080/ui/query.html?20230316_102958_00022_w7bqc
Splits: 11 total, 11 done (100.00%)
CPU Time: 0.0s total,    52 rows/s, 16.1KB/s, 70% active
Per Node: 0.0 parallelism,     1 rows/s,   466B/s
Parallelism: 0.1
Peak Memory: 716B
0.22 [1 rows, 313B] [4 rows/s, 1.37KB/s]

trino:tpch> SHOW CREATE TABLE t;
                                                                        Create Table
------------------------------------------------------------------------------------------------------------------------------------------------------------
 CREATE TABLE iceberg.tpch.t (
    a integer
 )
 WITH (
    format = 'ORC',
    format_version = 2,
    location = '/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294'
 )
(1 row)

$ find /var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/data
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/data/.20230316_102953_00021_w7bqc-b912a350-3c8d-4847-b8b8-9085d25c856d.orc.crc
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/data/20230316_102953_00021_w7bqc-b912a350-3c8d-4847-b8b8-9085d25c856d.orc
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata/.00001-f30b6abd-3491-4a4a-a5df-e046bf2ecff5.metadata.json.crc
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata/b6e1e849-607a-4d3a-8d23-d6b79ca19651-m0.avro
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata/.b6e1e849-607a-4d3a-8d23-d6b79ca19651-m0.avro.crc
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata/.00000-d34296ab-a0c2-4b2f-bdf6-7888a03b3415.metadata.json.crc
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata/snap-1422061323858791482-1-b6e1e849-607a-4d3a-8d23-d6b79ca19651.avro
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata/00001-f30b6abd-3491-4a4a-a5df-e046bf2ecff5.metadata.json
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata/.snap-1422061323858791482-1-b6e1e849-607a-4d3a-8d23-d6b79ca19651.avro.crc
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata/.20230316_102958_00022_w7bqc-0e2ca33b-3bf3-4cb3-bcab-60bd1e63362a.stats.crc
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata/20230316_102958_00022_w7bqc-0e2ca33b-3bf3-4cb3-bcab-60bd1e63362a.stats
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata/00000-d34296ab-a0c2-4b2f-bdf6-7888a03b3415.metadata.json

trino:tpch> DROP TABLE t;
DROP TABLE

After the DROP, one file still remains (+ its crc)

$ find /var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/data
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata/.20230316_102958_00022_w7bqc-0e2ca33b-3bf3-4cb3-bcab-60bd1e63362a.stats.crc
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata/20230316_102958_00022_w7bqc-0e2ca33b-3bf3-4cb3-bcab-60bd1e63362a.stats

[findepi]

cc @danielcweeks @nastra @alexjo2144 @grantatspothero

[haquem1]

Can I give this issue a shot?

[MehulBatra]

I would like to assign it to me and start working on it

[vakarisbk]

Hi, is there any update on this issue?

[findepi]

for REST catalog this probably will be fixed by apache/iceberg#9305

[ebyhr]

Confirmed the latest status. apache/iceberg#9305 is insufficient. Puffin files still remain if we execute ANALYZE statement or drop_extended_stats procedure.

In ANALYZE, the previous stats file path for the current snapshot is overwritten by the following code. Then, CatalogUtil.dropTableData can't delete the previous Puffin file because the latest metadata doesn't track the removed (logically) stats path.

trino/plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

Lines 2669 to 2671 in ddff01f

	transaction.updateStatistics()
	.setStatistics(snapshotId, statisticsFile)
	.commit();

In drop_extended_stats procedure, we call removeStatistics method. It just removes the stats info from metadata, so old Puffin files won't be deleted by Iceberg library.

trino/plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

Lines 1870 to 1872 in ddff01f

	UpdateStatistics updateStatistics = transaction.updateStatistics();
	for (StatisticsFile statisticsFile : icebergTable.statisticsFiles()) {
	updateStatistics.removeStatistics(statisticsFile.snapshotId());

Filed apache/iceberg#11876