[bug] Cannot perform table scan on V1 table

[kevinjqliu]

Apache Iceberg version

main (development)

Please describe the bug 🐞

While working with a V1 table, I noticed a few bugs which prevent table scan on V1 table.

1. Reading the manifest list defaults to V2
   

   iceberg-python/pyiceberg/manifest.py

   Line 635 in 620ad9f

   MANIFEST_LIST_FILE_SCHEMAS[DEFAULT_READ_VERSION],

2. Accessing fields not available in V1.
   

   iceberg-python/pyiceberg/table/__init__.py

   Line 1311 in 620ad9f

   if manifest.content == ManifestContent.DATA

   
   The content field is not available in V1, according to the spec.
   There are multiple places where something like this occurs.

Add a test to verify table scan on a V1 table

[Fokko]

@kevinjqliu Thanks for raising this. Can you elaborate on what you encountered when reading a V1 table? The Iceberg metadata is forward compatible, meaning we can turn any V1 table into a V2 (or V3) without issues.

The content field you mention will always be DATA in V1 (since there are no delete files). This can be solved easily with initial-default values. We do this in other places, such as sequence numbers.

It would be great to get a test that uncovers the issue so we can get this fixed :)

[kevinjqliu]

Added a reproducible test in #1483, i had to save the biglake iceberg table locally. please take a look

[kevinjqliu]

#1484 is a better, isolated test. It's using the minimal required schema for a v1 table manifest list.
In general, we should do this for all v1 schemas

[kevinjqliu]

Spark's V1 manifest list writer writes the optional added_rows_count field.

https://github.com/apache/iceberg/blob/fcd5dd932a21066d6127c94c50f3de43e8c2d80c/core/src/main/java/org/apache/iceberg/ManifestListWriter.java#L166-L167

https://github.com/apache/iceberg/blob/fcd5dd932a21066d6127c94c50f3de43e8c2d80c/core/src/main/java/org/apache/iceberg/V1Metadata.java#L31-L44