Third Generation Expression Engine

[joocer]

The expression engine has a few fundamental problems:

• Inconsistent representation, sometimes numpy, sometimes arrow, sometimes Python lists
• The Expression Engine is many layers deep with functions with similar names

Actions

• Create a new universal representation, we're going to wrap Arrow's buffers in our own array/table library - draken - initial testing has it between 3 and 10 times faster for comparisons.
• Write the core expression engine in Cython
• Precompile 'common' expressions to Cython (e.g. if we see name='john' and age > 17, have a Cython function something like compare_str_eq_and_int_gt(str_col, int_col, str_lit, int_lit, table)). We then apply that every time we have a similarly formed filter.

Our goal is to halve the execution time of the ClickBench benchmark.