feat(optimizer): Implement greedy join reordering

[desmondcheongzx]

Implements the Greedy Operator Ordering algorithm for join reordering.

With GreedyJoinOrderer, we can now transform join graphs back into logical plans with joins greedily reordered. Although we ultimately still want to apply DP-based join order algorithms, this greedy algorithm is still necessary when the number of join conditions is large and DP-based algorithms (which run in O(3^n), where n is the number of relations) are too expensive.

[codspeed-hq]

CodSpeed Performance Report

Merging #3538 will degrade performances by 49.57%

_{Comparing desmondcheongzx:goo-join-order (c77c3d1) with main (c3088d6)}

Summary

❌ 2 regressions
✅ 15 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	main	desmondcheongzx:goo-join-order	Change
❌	test_iter_rows_first_row[100 Small Files]	105.9 ms	210 ms	-49.57%
❌	test_show[100 Small Files]	15.8 ms	23.4 ms	-32.41%

[codecov]

Codecov Report

Attention: Patch coverage is 94.10029% with 20 lines in your changes missing coverage. Please review.

Project coverage is 77.72%. Comparing base (4567601) to head (c77c3d1).
Report is 7 commits behind head on main.

Files with missing lines	Patch %	Lines
...src/optimization/rules/reorder_joins/join_graph.rs	93.27%	15 Missing ⚠️
...imization/rules/reorder_joins/greedy_join_order.rs	95.23%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3538      +/-   ##
==========================================
+ Coverage   77.69%   77.72%   +0.03%     
==========================================
  Files         710      714       +4     
  Lines       86896    87389     +493     
==========================================
+ Hits        67513    67923     +410     
- Misses      19383    19466      +83     

Files with missing lines	Coverage Δ
src/common/scan-info/src/test/mod.rs	41.30% <100.00%> (+3.94%)	⬆️
src/daft-logical-plan/src/test/mod.rs	100.00% <100.00%> (ø)
src/daft-physical-plan/src/test/mod.rs	100.00% <100.00%> (ø)
...imization/rules/reorder_joins/greedy_join_order.rs	95.23% <95.23%> (ø)
...src/optimization/rules/reorder_joins/join_graph.rs	95.46% <93.27%> (-2.51%)	⬇️

... and 21 files with indirect coverage changes

[samster25]

I would strong avoid using the LogicalPlanRef as a hash key here. Whenever you do and insert or a look up the logical plan has to be hashed and then equity has to be computed. I would instead just assign each Node an incrementing ID or use the Arc::ptr as the ID

[samster25]

Alternatively you can also use something like a

struct GraphNodeKey {
  Id: usize,
  plan: LogicalPlanRef // ignored for hash and eq calcuations
}

[samster25]

do these need to pub(crate) or just local to the file pub(self) or nothing at all

[samster25]

I think this is doing too much (reordering and creating the logical plan) and isn't really that modular.

Instead what I would recommend is the following:

Trait JoinReorderer {
  fn reorder(&self, &mut graph: JoinGraph)
}

LogicalPlan -> JoinGraphBuilder -> JoinGraph -> JoinReorderer.reorder(JoinGraph) -> JoinGraph -> LogicalPlan

This way you get two things:

1. must easier to test that the JoinGraph roundtrip is working correctly
2. much easier to plug in a JoinReorderer (just implement that Trait)