[SPARK-52503][SQL][CONNECT] Fix drop when the input column is not existent

[zhengruifeng]

What changes were proposed in this pull request?

Fix drop when the input column is not existent

Why are the changes needed?

bugfix

import pyspark.sql.functions as F

df1 = spark.createDataFrame([("a", "b", "c") ], schema='colA string, colB string, colC string')

df2 = spark.createDataFrame([("c", "d", "") ], schema='colC string, colD string, colE string')

df3 = df1.join(df2, df1["colC"] == df2["colC"]).withColumn("colB", F.when(df1["colB"] == "b", F.concat(df1["colB"].cast("string"), F.lit("_newValue"))).otherwise(df1["colB"]))

df3
DataFrame[colA: string, colB: string, colC: string, colC: string, colD: string, colE: string]

df3.drop(df1["colB"])
DataFrame[colA: string, colC: string, colC: string, colD: string, colE: string]

df3 doesn't contains df1["colB"], so no columns should be dropped.

Does this PR introduce any user-facing change?

yes, bug fix

How was this patch tested?

added tests

Was this patch authored or co-authored using generative AI tooling?

no

[cloud-fan]

how is this implemented? I see we pass throws = true when calling resolveExpression

[zhengruifeng]

if it cannot find the plan id in the plan, then it fails;
if the plan id is found, but cannot resolve it, then return None