Custom Indexes in Apache DataFusion

[P1ntas]

Hello,
I'm currently working on my dissertation, exploring how SIMD-optimized index data structures can enhance performance in column-oriented databases, specifically targeting analytical workloads. During my research, DataFusion stood out due to being based on Apache Arrow and its suitability for analytical queries. I want to use it to implement and benchmark my proposed solutions.
I would like to know if it is feasible to implement custom indexes within DataFusion. The help of people already experinced with this technology would be great to help me direct my focus.
Thanks in advance for your help!

[alamb]

Hi @P1ntas 👋 it is great to see you!

Here is one example
https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/advanced_parquet_index.rs

But that relies on parquet

In general, the TableProvider interface gives you everything you would need for using a custom index -- you get the filters and projection and then the provider needs to apply that to the index

Depending on what your index is doing, you might find https://docs.rs/datafusion/latest/datafusion/physical_optimizer/pruning/struct.PruningPredicate.html helpful too