Andrew Lamb Weekly-ish Open Source plan - 2025-11-03

[alamb]

This is my weekly plan, mostly for my own organizational needs (as I am dropping too many things). I am making it public in the hopes that helps others to see what I am working on -- also I spend so much time in github the interface is very familiar to me and I can cross link all the issues I am working

(it is also my excuse as to why I haven't reviewed many good looking PRs)

I will update this list as I run into more things

PR review queue (rough order)

• Normalize partitioned and flat object listing #18146
• add try_new_with_length constructor to FixedSizeList arrow-rs#8624
• CI: add clippy::needless_pass_by_value rule #18468
• Add options to control various aspects of Parquet metadata decoding arrow-rs#8763
• Consolidate udf examples (#18142) #18493
• Fix: ViewType gc on huge batch would produce bad output arrow-rs#8694
• Respect execution timezone in to_timestamp and related functions #18025
• Add options to control various aspects of Parquet metadata decoding arrow-rs#8763
• General virtual columns support + row numbers as a first use-case arrow-rs#8715
• Implement a Vec<RecordBatch> wrapper for pyarrow.Table convenience arrow-rs#8790
• Refactor distinct aggregate implementations to use common buffer #18348
• Refactor create_hashes to accept array references #18448
• 8762
• #17801 Improve nullability reporting of case expressions #17813
• [Variant] Enforce shredded-type validation in shred_variant arrow-rs#8796 (review)
• Test datafusion with new sqlparser: Eliminating whitespace from the parser logic datafusion-sqlparser-rs#2076 (comment)
• Refactor InListExpr to store arrays and support structs #18449
• Refactor state management in HashJoinExec and use CASE expressions for more precise filters #18451
• Support Arrow IPC Stream Files #18457
• Add FilterBuilder::is_optimize_beneficial arrow-rs#8782
• Add merge and merge_n algorithms arrow-rs#8753
• feat: support named variables & defaults for CREATE FUNCTION #18450
• Add relation planner extension support #17843

What am I personally working on now

• DataFusion meetup prep next week: DISCUSSION: DataFusion Meetup in Boston, USA - Nov 12, 2025 #16703
• Internal error: Physical input schema should be the same as the one converted from logical input schema. #18337 (and related fallout)
• Parquet push decoder: Rewrite ParquetRecordBatchStream in terms of the PushDecoder arrow-rs#8159

Projects I am supporting actively (high on my priority list)

• Next DF release: Release DataFusion 51.0.0 (Nov 2025) #17558 with @xudong963
• Adaptive parquet predicate pushdown evaluation faster with @hhhizzz [Parquet] Adaptive Parquet Predicate Pushdown arrow-rs#8733
• Low level arrow bit manipulation fast with @rluvaton feat: add apply_unary_op and apply_binary_op bitwise operations arrow-rs#8619
• DataFusion object store requests go faster with @BlakeOrth [EPIC] ListingTable object store usage improvements #17214
• Release object store 0.13.0: Release object store 0.13.0 (breaking) - Target Nov 2025 arrow-rs-object-store#367
• Improve DataFusion ClickBench performance: [EPIC] Make DataFusion the top of the ClickBench Parquet leaderboard #18489

Projects on my backlog

These are ones I would like to support but don't have the capacity at the moment to push, in relative order

• row numbers in parquet [Parquet] Support file row number in Parquet reader arrow-rs#7299 with @jkylling and @vustef
• Help integrate Variant with @friendlymatthew [EPIC] Support VARIANT type for unstructured data #16116
• Epic: Join Order Enumeration #18249 from @NGA-TRAN

PRs that look great but need a thorough review (looking for help here 🎣 from anyone else)

• Examples of extending SQL syntax #17824 from the @theirix
• external tables for multiple locations: feat(cli): support external tables on multiple locations #17702
• relation extension planner: Add relation planner extension support #17843
• writing REE arrays to parquet: Support writing RunEndEncoded as Parquet arrow-rs#8069