[SPARK-50694][SQL] Support withColumns / withColumnsRenamed in subqueries

[ueshin]

What changes were proposed in this pull request?

Supports withColumns / withColumnsRenamed in subqueries.

Why are the changes needed?

When the query is used as a subquery by adding col.outer(), withColumns or withColumnsRenamed doesn't work because they need analyzed plans.

Does this PR introduce any user-facing change?

Yes, those APIs are available in subqueries.

How was this patch tested?

Added the related tests.

Was this patch authored or co-authored using generative AI tooling?

No.

[HyukjinKwon]

cc @cloud-fan too

[cloud-fan]

I don't think we need new logical plans. See how we implement a similar feature for SQL pipe: 68be1da

[cloud-fan]

We can reuse Project with UnresolvedStarExceptOrReplace

[cloud-fan]

oh, df.withColumn can append or replace columns, we should probably extend UnresolvedStarExceptOrReplace for this case.

[cloud-fan]

cc @dtenedor

[ueshin]

There seem to be some semantic differences even with withColumnsRenamed:

• If the specified column names are missing in the df, df.withColumnsRenamed ignores whereas UnresolvedStarExceptOrReplace throws an exception.
• df.withColumnsRenamed respects the argument order, e.g.,

  test("SPARK-46260: withColumnsRenamed should respect the Map ordering") {
    val df = spark.range(10).toDF()
    assert(df.withColumnsRenamed(ListMap("id" -> "a", "a" -> "b")).columns === Array("b"))
    assert(df.withColumnsRenamed(ListMap("a" -> "b", "id" -> "a")).columns === Array("a"))
  }
  

  whereas UnresolvedStarExceptOrReplace throws an exception.

I guess we should keep the new plans to be safer?

[cloud-fan]

I see. One question is shall we add new logical plans or new star-like expressions? star-like expressions are more flexible as they work for function input as well (e.g. struct(*)). And we might be able to unify them with the SQL pipe one with extra flags.

[ueshin]

Moved to the star-like expressions.