feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations

[duongcongtoai]

Which issue does this PR close?

• Closes Supporting planning (binding) Nested correlated subquery error with a depth exceeding 1 #19816

Original PR/discussions: #16186

• Partially solve Support Nested correlated subquery error with a depth exceeding 1 #15558 and unblock feat: rewrite subquery into dependent join logical plan #16016

There were some discussion going on regarding handle subquery with depth aware at the planning stage, which is a nice thing to have, but until we implement something like that, we cannot continue implement query decorrelation. But i realize that we can add some minor change to how we plan the subqueries so at least no error is thrown because of ambiguous schema as in #15558:

• PlannerContext maintains an optional outer schema, we just need to replace this field with a stack of outer schema

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

[kosiew]

@duongcongtoai

You'll get more helpful review comments if you tell us more about your PR

[alamb]

Thanks @duongcongtoai and @duongcongtoai

It turns out I got a ping yesterday from Andy Pavlo and his student @yliang412 about the status of subquery support)

I just filed a ticket to track just the planning aspect of this query (which I think this PR handles)

• Supporting planning (binding) Nested correlated subquery error with a depth exceeding 1 #19816

[alamb]

I will try and give this a look shortly

[alamb]

I do think this is the key change -- that rather than having a single outer query schema, we need to have a stack, one for each in the relation.

However it seems like we shouldn't need both outer_query_schema as well a stack of them -- outer_query_schemas should always be enough -- maybe you can add push and pop methods that add/remove from the schema stack

One thought I had while looking at this PR is that a more logical structure would be for each PlannerContext to have a reference to its parent -- something like

pub struct PlannerContext<'a> {
  /// When planning a subquery, the context of the outer query
  parent_context: Option<&'a PlannerContext>
  ...
}

And remove the explicit outer_query_schema

[alamb]

Thank you for this @duongcongtoai and @kosiew

Our friends at CMU were just asking about this so it was great to see a PR already started

I left some comments. My biggest ones are:

1. Can we find a way to unify the schema stacks (rather than having both a field and a stack)?
2. Can we restrict this PR to just planning (aka no changes to the optimizer or other stages) -- maybe with tests focused only on planning, such as

   datafusion/datafusion/sql/tests/sql_integration.rs

   Lines 4380 to 4398 in 4e0161d

   	#[test]
   	fn test_ambiguous_column_references_with_in_using_join() {
   	let sql = "select p1.id, p1.age, p2.id
   	from person as p1
   	INNER JOIN person as p2
   	using(id)";
   	let plan = logical_plan(sql).unwrap();
   	assert_snapshot!(
   	plan,
   	@r"
   	Projection: p1.id, p1.age, p2.id
   	Inner Join: Using p1.id = p2.id
   	SubqueryAlias: p1
   	TableScan: person
   	SubqueryAlias: p2
   	TableScan: person
   	"
   	);
   	}