Dataframe with_column and with_column_renamed performance improvements

[Omega359]

Which issue does this PR close?

• Part of Perf: Dataframe with_column and with_column_renamed are slow #14563, related to Early exit on column normalisation to improve DataFrame performance #14636

with_column_10          time:   [3.6602 ms 3.8457 ms 4.2130 ms]
with_column_100         time:   [34.874 ms 35.979 ms 38.018 ms]
with_column_200         time:   [183.49 ms 187.29 ms 191.65 ms]

If there is any dataframe experts here I would love a review of my assumptions. As noted in #14563 (comment)

My thinking for the changes in my current branch was that any 'new' Expr (parameter to with_column, with_column_renamed, etc) would go through the normalization, everything else I would like to think would already have been normalized or else how would it be in the DataFrame? What worries me is that I don't know if that assumption is correct or not.

Rationale for this change

Improve performance for with_column and with_column_renamed dataframe functions.

What changes are included in this PR?

Code

Are these changes tested?

Existing tests

Are there any user-facing changes?

No.

[Omega359]

After spending more time reviewing the dataframe and logical plan code I have a feeling that my assumption is in fact not correct and that a dataframe can indeed have a plan that is not normalized/columnized prior to with_column being called. Joins, window, aggregate, are possible examples.

[timsaucer]

I suspect you're right about that assumption not being correct. I've dug through a bit, but I'd probably need to write up a unit test to verify.

[Omega359]

I've made some changes locally where I test to see if the existing plan is a projection but I realized that I can't just rely on that either as the plan could possibly have been manually made then a DataFrame wrapped around it and the with_column function called.

For my approach to work I would need a way to strongly guarantee that the last projection that was made was done via the project(..) function in the builder where the normalization/columnization is guaranteed to have happened. I'm not sure right now how to do that

[Omega359]

with_column_10          time:   [6.1112 ms 6.2616 ms 6.4226 ms]
                        change: [+18.276% +23.739% +29.703%] (p = 0.00 < 0.05)
with_column_100         time:   [41.379 ms 54.353 ms 66.683 ms]
                        change: [+18.326% +33.842% +55.521%] (p = 0.00 < 0.05)
with_column_200         time:   [198.59 ms 209.45 ms 224.33 ms]
                        change: [-1.0194% +5.1906% +12.830%] (p = 0.20 > 0.05)
with_column_500         time:   [3.5914 s 3.7758 s 4.0800 s]
                        change: [-13.580% -6.5454% +2.3214%] (p = 0.16 > 0.05)
                        No change in performance detected.

[Omega359]

This should be ready for review.

[alamb]

Thanks @Omega359 -- this makes sense to me

I think it would be nice to avoid a new pub function and add a few more comments, but I also don't think that is required

[alamb]

🎉

[alamb]

Could you add some additional comments here about what circumstances permit validation to be skipped?

[Omega359]

Updated, please review text when you have a chance.

[alamb]

Thanks again @Omega359