EPIC: spark connect

[universalmind303]

spark connect

distributed execution

for distributed execution we need a ray runner that we can call from rust

• create rust based shim around our existing python ray runner

We might need this?

• move DaftContext into rust (this one should be relatively easy)

compatibility/interop

some of the text based methods (printSchema, show, explain) should have a spark compatibile output.

• modify the to_comfy_table to be able to output a spark compatible df output.
• alternative display implementation for Schema that matches spark's
• create a new TreeDisplay implementation that somewhat matches spark's plans

pyspark.sql.DataFrame

• agg
• alias
• approxQuantile
• cache
• checkpoint
• coalesce
• colRegex
• collect
• corr
• count
• cov
• createGlobalTempView
• createOrReplaceGlobalTempView
• createOrReplaceTempView
• createTempView
• crossJoin
• crosstab
• cube
• describe
• distinct
• drop
• dropDuplicates
• dropDuplicatesWithinWatermark
• dropna
• exceptAll
• explain
• fillna
• filter
• first
• foreach
• foreachPartition
• TODO: add rest of items.

pyspark.sql.Catalog

TODO (I don't think this is stabilized in spark connect yet)

pyspark.sql.functions

see https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/functions.html for list of functions

UDFS

spark UDF's should be mappable to our UDF's. They use a very similar pickling approach, and we'll likely just need to use their deserializer to deserialize them back into python. Likely a bit more discovery needed.

UX/DX

• connect: expose a single executable for spark connect.  #3551
• move daft.daft.spark_connect into a dedicated module #3497
• connect: make daft spark an optional, opt-in feature. #3582

Documentation

• Quick start guide for spark connect via daft (daft connect)
• Distributed computing guide for daft connect
  • this should include a guide for how to set up a cluster and connect to it via spark.

Issue Tracking

• connect: df.explain() #3577
• connect: expose a single executable for spark connect.  #3551
• connect: unify read.rs, write.rs #3550
• spark connect fails when performing a .show() #3498
• move daft.daft.spark_connect into a dedicated module #3497
• daft-connect: support CastToType::TypeStr(s) #3486
• daft connect: add more sort tests #3424
• connect: implement CSV null value handling #3609

Upstream Spark issues

• https://issues.apache.org/jira/browse/SPARK-50627
• https://issues.apache.org/jira/browse/SPARK-50626

[jaychia]

WRT to the catalogs, @universalmind303 what do you think of starting to unify around the DaftMetaCatalog that I introduced in #3036?

I think we have a few competing standards atm (including the SQLCatalog). It could be good to start having a catalog abstraction that can be shared across our different frontends

[universalmind303]

WRT to the catalogs, @universalmind303 what do you think of starting to unify around the DaftMetaCatalog that I introduced in #3036?

I think we have a few competing standards atm (including the SQLCatalog). It could be good to start having a catalog abstraction that can be shared across our different frontends

yes, that is something I want to do and have been thinking about. I'll open up an issue to unify daft.catalog and daft.sql.catalog as well