Polars: Chapter 14 - User-Defined Functions

[peter-gy]

📝 Summary

The core idea of this chapter is to demonstrate realistic scenarios where the flexibility of user-defined functions outweighs potential performance drawbacks, while simultaneously providing an overview of relevant Polars expressions for UDFs and utilizing the interactive Marimo environment.

In the "🚀 Higher-performance UDFs" section I introduced concepts like NumPy ufuncs and generalized ufuncs but did not go into huge detail on purpose, as there is a dedicated "NumPy functions" chapter for that in the course outline #40.

Chapter outline:

• ⚖️ The Cost of UDFs
• 📊 Project Overview
• 🔂 Element-Wise UDFs
• 📦 Batch-Wise UDFs
• ⚙️ Row-Wise UDFs
• 🚀 Higher-performance UDFs
• ⏱️ Quantifying the Overhead

You can try this notebook by executing:

uvx marimo run https://github.com/peter-gy/learn/blob/33b7a6233c066885fc3fb127c5e2f2f4d769c195/polars/14_user_defined_functions.py --sandbox

📋 Checklist

• I have included package dependencies in the notebook file using --sandbox
• Keep language direct and simple.

[mscolnick]

wow, this is such a great notebook. great stuff @peter-gy

[Haleshot]

Wow; it was a great experience to review the notebook! Really informative, packed with native UI elements to play around with and understand the effects on relevant plots.

Right from starting the notebook (as to why UDFs are better for certain reasons but have certain drawbacks in terms of performance) to giving info on how combining it with Numba can lead to better efficiency at the end; this notebook proved to be really informational.

Thanks a ton for this high-quality notebook contribution 😃