feat: nan_value_counts support #907

feniljain · 2025-01-22T20:16:02Z

Issue

Fixes #417

We compute upper and lower bounds by relying on parquet statistics, but those statistics don't provide nan_value_count, so we have to implement it in library itself when arrow record batches are received.
We keep track of it at ParquetWriter level cause write can be called multiple times .
Added new assert for nan_val_count in test_parquet_writer itself.

liurenjie1024

Thanks @feniljain for this pr! Left some comments to resolve.

liurenjie1024 · 2025-01-24T09:05:32Z

crates/iceberg/src/writer/file_writer/parquet_writer.rs

+            let dt = col.data_type();
+
+            let nan_val_cnt: u64 = match dt {
+                DataType::Float32 => {


I think we also need to add float64?

Also this is incorrect, it ignored nested primitive type.

feat: nan_value_counts support

7e76ade

feniljain force-pushed the feat-nan-value-counts branch from 281daa1 to 7e76ade Compare January 22, 2025 20:17

liurenjie1024 reviewed Jan 24, 2025

View reviewed changes