Epic: Statistics improvements

[alamb]

Is your feature request related to a problem or challenge?

We would like to use "statistics" in our project for transformations that rely on the statisics being "correct" (e.g. that the there are no values outside the min and max range).

DataFusion has several optimizations like this too that rely on statistics being correct such as skipping file scans with limits such as in https://github.com/apache/arrow-datafusion/blob/e54894c39202815b14d9e7eae58f64d3a269c165/datafusion/core/src/datasource/statistics.rs#L34-L33. There are also suggestions of additional such optimizations like #6672

However the current Statistics code seems to make it hard to manage the 'are the statistics exact and can they be guaranteed for transformations' (@crepererum noted this quite some time ago on #5613). This has recently lead to several bugs such as

• Fix incorrect results in COUNT(*) queries with LIMIT #8049
• Bug-fix in Filter and Limit statistics #8094

We would like to make it clearer what is known and what is an estimate is know (e.g. the min/max of row counts may be known, but the actual value may be an estimate after a filter). This is described in more detail on #8078

As we began exploring this concept we ran into several issues with Statistics and I think it is getting big enough to warrant its own tracking epic

Related items

• Show statistics min and max in explain #8110
• Show Statistics in explain Verbose without enabling show statistics #8111
• ParquetExec::statistics() does not read statistics for many column types (like timstamps, strings, etc) #8295
• Consolidate statistics aggregation #8229
• Improve statistics test coverage #8228
• Introduce a way to represent constrained statistics / bounds on values in Statistics #8078
• Statistics::total_byte_size does not account for projection in FileScanConfig::with_projection #14936
• Make default filter selectivity estimate configurable #8133
• Queries that lost statistics or their statistics become inexact #8099
• Use file statistics in query planning to avoid sorting when unecessary #7490
• ParquetExec::statistics::is_exact likely wrong/misunderstood #5614
• Statistics::is_exact semantics #5613
• Possible bug in Parquet pruning code? #8309
• Bug: fix case where num_rows and total_byte_size are not defined (stat should be None instead of Some(0)) #2976
• Support parquet statistics for struct columns #8334
• Parquet pruning will be incorrect if field names are repeated #8335
• Merging Statistics is slow when sum statistic is present #15809
• Parquet Predicate Pushdown Does Not Handle Type Coercion #7925
• Parquet Statistics Pruning Ignores ColumnOrder, resulting in potentially incorrect statistics #8342
• Merging Statistics is slow when sum statistic is present #15809 : manipulating the statistics (e.g. merging files to get partitions statistics) is slow

Pruning Improvements (maybe should be its own epic)

• Don't error on unknown column when pruning if predicate can still be proven false #7869
• Support general pruning based on <col> = 'const' in PruningPredicate #8376

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

This is somewhat related