[3.2][Spark] Update OptimizeGeneratedSuite to apply constant folding (#…

…3141)

The following change in Spark master broke tests in
`OptimizeGeneratedColumnSuite`:
apache/spark@7974811

It added an execution of the `ConstantFolding` rule after
`PrepareDeltaScan`, causing constant expressions in filters on generated
columns to be simplified, which `OptimizeGeneratedColumnSuite` heavily
used.

This change:
- updates the expected results in `OptimizeGeneratedColumnSuite` to
simplify constant expressions
- adds a pass of `ConstantFolding` after `PrepareDeltaScan` so that
Delta on spark 3.5 behaves the same as Delta on spark master.

Updated tests

spark/src/main/scala/io/delta/sql/DeltaSparkSessionExtension.scala

@@ -25,6 +25,7 @@ import org.apache.spark.sql.delta.stats.PrepareDeltaScan
 import io.delta.sql.parser.DeltaSqlParser
 
 import org.apache.spark.sql.SparkSessionExtensions
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
 import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
 import org.apache.spark.sql.catalyst.rules.Rule
 import org.apache.spark.sql.delta.PreprocessTimeTravel
@@ -128,12 +129,12 @@ class DeltaSparkSessionExtension extends (SparkSessionExtensions => Unit) {
 
     extensions.injectPlanNormalizationRule { _ => GenerateRowIDs }
 
-    // We don't use `injectOptimizerRule` here as we won't want to apply further optimizations after
-    // `PrepareDeltaScan`.
-    // For example, `ConstantFolding` will break unit tests in `OptimizeGeneratedColumnSuite`.
     extensions.injectPreCBORule { session =>
       new PrepareDeltaScan(session)
     }
+    // Fold constants that may have been introduced by PrepareDeltaScan. This is only useful with
+    // Spark 3.5 as later versions apply constant folding after pre-CBO rules.
+    extensions.injectPreCBORule { _ => ConstantFolding }
 
     // Add skip row column and filter.
     extensions.injectPlannerStrategy(PreprocessTableWithDVsStrategy)