Does Sail currently support a mechanism similar to Spark’s native External Shuffle Service (ESS)?

[weixiuli]

Does Sail currently support a shuffle management mechanism similar to Spark’s native External Shuffle Service (ESS)? In large-scale shuffle scenarios, how does Sail manage and persist shuffle data, especially when shuffle data exceeds memory capacity or when worker nodes fail?

[weixiuli]

@linhr Thanks.

[linhr]

Currently, Sail supports streaming shuffle, meaning that the data being exchanged among workers stay in memory.

We have a plan to support blocking shuffle so that the data can be persisted and reused when a worker fails. We will likely start with object storage (AWS S3 or similar services supported by the object_store library) for shuffle data. We feel that external shuffle services are of less importance in cloud-native environments if we design the shuffle mechanism carefully. But we would love to hear use cases to shape our roadmap.

[weixiuli]

Hi @linhr ,
Sorry to bother you. I have been following Sail / LakeSail recently and have some technical questions I would like to ask you.
If convenient, would you mind sending your WeChat contact information to my email address: weixiuliysu@gmail.com?
Thank you very much for your help.