Best way to swap out Rayon for chili

[yonas]

Awesome work! What's the easiest way for non-experts to take advantage of chili if they know their favorite Rust project is using Rayon?

[dragostis]

The benchmarks have the same example running on both chili and Rayon.

The main difference right now is that you need to create a ThreadPool and pass a &mut Scope that you can then run join on.

[impactdni2]

A major use-case for rayon, is the roughly zero-setup parallelization of collection processing (.par_iter()). Any plans to include something like that directly in Chili?

[dragostis]

@impactdni2, the reason why I focused on join specifically was that parallel iterators have a means of getting good performance by tuning with_min_len argument.

If heartbeat scheduling proves to be useful for parallel iterators, maybe the best course of action is to try and integrate both this and the join function into Rayon proper.

[cuviper]

See also rayon-rs/rayon#1188 -- I think it would be quite interesting to explore such changes in rayon-core!

[impactdni2]

@dragostis Completely understand, and I'll definitely look at moving parts of my application that decompose nicely into join to use chili rather than Rayon. The main pain-point/use-case for me is that in a number of places, I have a list of non-homogeneous jobs that make it very difficult to accurately predict if the overhead of rayon is worth it to process in parallel. Some of the jobs might take 500µs and get a nice speedup from rayon, and some of the jobs might end up only taking 2µs, making the overhead on rayon end up not giving much of a speedup. Was hoping the lower overhead here might solve that :)

[dragostis]

Closing this in favor of #12. The plan is to implement the entire rayon-core API in order to get parallel iterators working.