Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I want to test out the parquet filter pushdown on real datasets using datafusion-cli so we can enable it by default -- #3463
To do so I want to both understand what the current setting of the pushdown flags are as well as change them statement by statement.
Describe the solution you'd like
Use the ConfigOptions to control parquet scanning options rather than anothr structure
Among other things that will allow the parquet settings to appear here as well as be controlled by environment variable in datafusion-cli.
It will also allow this feature to be turned off if we find an issue
❯ show all;
+-------------------------------------------------+---------+
| name | setting |
+-------------------------------------------------+---------+
| datafusion.execution.time_zone | UTC |
| datafusion.optimizer.skip_failed_rules | true |
| datafusion.explain.logical_plan_only | false |
| datafusion.optimizer.filter_null_join_keys | false |
| datafusion.explain.physical_plan_only | false |
| datafusion.execution.batch_size | 8192 |
| datafusion.execution.coalesce_batches | true |
| datafusion.execution.coalesce_target_batch_size | 4096 |
+-------------------------------------------------+---------+Describe alternatives you've considered
Can just do it programmatically
Additional context
See #3463