[SPARK-50830][SQL] Return single-pass result as the dual run analysis…

… result ### What changes were proposed in this pull request? Right now fixed-point resolved plan is return as a result of analysis of the unresolved plan. In this task we add a flag in order to guard returning of the single-pass result in specific cases so it can be reverted more easily + return single-pass resolver result as the result of the dual-run analysis. ### Why are the changes needed? Goal is to have single-pass resolver result as the global one (analysis result of an unresolved plan). We are introducing it now but also adding a flag so it can be reverted easily. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing tests with the `ANALYZER_DUAL_RUN_LEGACY_AND_SINGLE_PASS_RESOLVER` value set to true. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#49509 from mihailoale-db/mihailo-aleksic_data/mihailoale-db/singlepassresultflag. Authored-by: mihailoale-db <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
beliefer · Jan 16, 2025 · 6ebed5b · 6ebed5b
1 parent d12bb23
commit 6ebed5b
Show file tree

Hide file tree

Showing 2 changed files with 17 additions and 1 deletion.
diff --git a/...alyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/HybridAnalyzer.scala b/...alyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/HybridAnalyzer.scala
@@ -163,7 +163,11 @@ class HybridAnalyzer(
             throw singlePassEx
           case None =>
             validateLogicalPlans(fixedPointResult.get, singlePassResult.get)
-            fixedPointResult.get
+            if (conf.getConf(SQLConf.ANALYZER_DUAL_RUN_RETURN_SINGLE_PASS_RESULT)) {
+              singlePassResult.get
+            } else {
+              fixedPointResult.get
+            }
         }
     }
   }

diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -276,6 +276,18 @@ object SQLConf {
       .booleanConf
       .createWithDefault(false)
 
+  val ANALYZER_DUAL_RUN_RETURN_SINGLE_PASS_RESULT =
+    buildConf("spark.sql.analyzer.singlePassResolver.returnSinglePassResultInDualRun")
+      .internal()
+      .doc(
+        "When true, return the result of the single-pass resolver as the result of the dual run " +
+        "analysis (which is used if the ANALYZER_DUAL_RUN_LEGACY_AND_SINGLE_PASS_RESOLVER flag " +
+        "value is true). Otherwise, return the result of the fixed-point analyzer."
+      )
+      .version("4.0.0")
+      .booleanConf
+      .createWithDefault(Utils.isTesting)
+
   val ANALYZER_SINGLE_PASS_RESOLVER_VALIDATION_ENABLED =
     buildConf("spark.sql.analyzer.singlePassResolver.validationEnabled")
       .internal()