Please leave a comment and mark on the TODO tests in this issue if you want to contribute
Backend
VL(Velox)
Bug Description
These suites were commented out in #11512 with TODO markers indicating they need to be fixed and re-enabled.
Context
The test suites below have been disabled due to various failures. Each table shows the status for both Spark 4.0 and 4.1:
- 🔴 = Suite is commented out (disabled)
- 🟢 = Suite is enabled
The failure count (when available) is shown in the status column.
org.apache.spark.sql
| Suite |
Spark 4.0 |
Spark 4.1 |
Owner |
Comments |
GlutenDataFrameSubquerySuite |
🟢 |
🟢 |
@baibaichen |
Fixed in #11727 |
GlutenExplainSuite |
🔴 (1 failures) |
🔴 (1 failures) |
@Surbhi-Vijay |
|
GlutenJoinHintSuite |
🔴 (1 failure) |
🔴 (1 failure) |
|
Discovered in #11800 |
GlutenLogQuerySuite |
🔴 (2 failures) |
🔴 (2 failures) |
@kapilks |
|
GlutenRandomDataGeneratorSuite |
🟢 |
🔴 (232 failures) |
|
Discovered in #11800 |
GlutenSetCommandSuite |
�� (1 failures) |
🔴 (1 failures) |
@Surbhi-Vijay |
|
GlutenSingleLevelAggregateHashMapSuite |
🔴 (1 failures) |
🔴 (1 failures) |
@kapilks |
|
GlutenSparkSessionJobTaggingAndCancellationSuite |
🔴 (1 failures) |
🔴 (1 failures) |
|
|
GlutenTwoLevelAggregateHashMapSuite |
🔴 (1 failures) |
🔴 (1 failures) |
@kapilks |
|
GlutenTwoLevelAggregateHashMapWithVectorizedMapSuite |
🔴 (1 failures) |
🔴 (1 failures) |
@kapilks |
|
GlutenVariantEndToEndSuite |
🟢 |
🟢 |
@baibaichen |
Fixed in #11726 |
GlutenVariantShreddingSuite |
🟢 |
🟢 |
@baibaichen |
Fixed in #11726 |
GlutenXmlFunctionsSuite |
🟢 |
🟢 |
@baibaichen |
Fixed in #11725 |
GlutenTPCDSV1_4_PlanStabilitySuite |
🟢 |
🟢 |
@baibaichen |
Fixed in #11799 |
GlutenTPCDSV1_4_PlanStabilityWithStatsSuite |
🟢 |
🟢 |
@baibaichen |
Fixed in #11799 |
GlutenTPCDSV2_7_PlanStabilitySuite |
🟢 |
🟢 |
@baibaichen |
Fixed in #11799 |
GlutenTPCDSV2_7_PlanStabilityWithStatsSuite |
🟢 |
🟢 |
@baibaichen |
Fixed in #11799 |
GlutenTPCDSModifiedPlanStabilitySuite |
🟢 |
🟢 |
@baibaichen |
Fixed in #11799 |
GlutenTPCDSModifiedPlanStabilityWithStatsSuite |
🟢 |
🟢 |
@baibaichen |
Fixed in #11799 |
GlutenTPCHPlanStabilitySuite |
🟢 |
🟢 |
@baibaichen |
Fixed in #11799 |
catalyst.expressions
| Suite |
Spark 4.0 |
Spark 4.1 |
Owner |
Comments |
GlutenCastWithAnsiOnSuite |
🔴 (4 failures) |
🔴 (10 failures) |
|
Discovered in #11800 |
GlutenCollationRegexpExpressionsSuite |
🟢 |
🔴 (1 failure) |
|
Discovered in #11800 |
GlutenExpressionEvalHelperSuite |
🔴 (2 failures) |
🔴 (2 failures) |
|
Discovered in #11800 |
GlutenObjectExpressionsSuite |
🔴 (7 failures) |
🔴 (7 failures) |
|
Discovered in #11800 |
GlutenOrderingSuite |
🟢 |
🔴 (2 failures) |
|
Discovered in #11800 |
GlutenScalaUDFSuite |
🔴 (1 failure) |
🔴 (1 failure) |
|
Discovered in #11800 |
GlutenToPrettyStringSuite |
🔴 (1 failure) |
🔴 (1 failure) |
|
Discovered in #11800 |
GlutenXmlExpressionsSuite |
🟢 |
🟢 |
@baibaichen |
Fixed in #11580 |
connector
| Suite |
Spark 4.0 |
Spark 4.1 |
Owner |
Comments |
GlutenGroupBasedUpdateTableSuite |
🔴 (1 failures) |
🔴 (1 failures) |
@Surbhi-Vijay |
|
GlutenMergeIntoDataFrameSuite |
🔴 (1 failures) |
🔴 (1 failures) |
|
|
execution
| Suite |
Spark 4.0 |
Spark 4.1 |
Owner |
Comments |
GlutenCoalesceShufflePartitionsSuite |
🔴 (ABORTED) |
🔴 (ABORTED) |
|
Discovered in #11800 |
GlutenColumnarRulesSuite |
🟢 |
🔴 (1 failure) |
|
Discovered in #11800 |
GlutenDataSourceScanExecRedactionSuite |
🔴 (2 failures) |
🔴 (2 failures) |
|
Discovered in #11800 |
GlutenDataSourceV2ScanExecRedactionSuite |
🔴 (2 failures) |
🔴 (2 failures) |
|
Discovered in #11800 |
GlutenExternalAppendOnlyUnsafeRowArraySuite |
🔴 (14 failures) |
🔴 (14 failures) |
|
Discovered in #11800 |
GlutenHiveResultSuite |
🟢 |
🔴 (1 failure) |
|
Discovered in #11800 |
GlutenInsertSortForLimitAndOffsetSuite |
🔴 (6 failures) |
🔴 (6 failures) |
|
|
GlutenLogicalPlanTagInSparkPlanSuite |
🔴 (ABORTED) |
🔴 (ABORTED) |
|
Discovered in #11800 |
GlutenPlannerSuite |
🔴 (1 failures) |
🔴 (1 failures) |
|
|
GlutenProjectedOrderingAndPartitioningSuite |
🔴 (6 failures) |
🔴 (6 failures) |
|
Discovered in #11800 |
GlutenRemoveRedundantProjectsSuite |
🔴 (14 failures) |
🔴 (14 failures) |
|
Discovered in #11800 |
GlutenRemoveRedundantSortsSuite |
🔴 (1 failures) |
🔴 (1 failures) |
|
|
GlutenSQLExecutionSuite |
🔴 (1 failure) |
🔴 (1 failure) |
|
Discovered in #11800 |
GlutenSQLJsonProtocolSuite |
🔴 (1 failure) |
🔴 (1 failure) |
|
Discovered in #11800 |
GlutenShufflePartitionsUtilSuite |
🔴 (1 failure) |
🔴 (1 failure) |
|
Discovered in #11800 |
GlutenSimpleSQLViewSuite |
🔴 (1 failure) |
🔴 (2 failures) |
|
Discovered in #11800 |
GlutenSparkPlanSuite |
🔴 (1 failures) |
🔴 (1 failures) |
@Surbhi-Vijay |
|
GlutenUnsafeRowSerializerSuite |
🔴 (1 failure) |
🔴 (1 failure) |
|
Discovered in #11800 |
GlutenWholeStageCodegenSparkSubmitSuite |
🔴 (1 failures) |
🔴 (1 failures) |
|
|
GlutenWholeStageCodegenSuite |
🔴 (24 failures) |
🔴 (24 failures) |
@Surbhi-Vijay |
|
execution.joins
| Suite |
Spark 4.0 |
Spark 4.1 |
Owner |
Comments |
GlutenSingleJoinSuite |
🟢 |
🟢 |
@baibaichen |
Fixed in #11577 |
execution.datasources.parquet
| Suite |
Spark 4.0 |
Spark 4.1 |
Owner |
Comments |
GlutenParquetCommitterSuite |
🔴 (spark.test.home) |
🔴 (spark.test.home) |
|
Discovered in #11800 |
GlutenParquetTypeWideningSuite |
🔴 (74 failures) |
🔴 (74 failures) |
@baibaichen |
track in #11683 |
GlutenParquetVariantShreddingSuite |
🟢 |
🟢 |
@baibaichen |
Fixed in #11726 |
execution.datasources.text
| Suite |
Spark 4.0 |
Spark 4.1 |
Owner |
Comments |
GlutenWholeTextFileV1Suite |
🔴 (1 failures) |
🔴 (1 failures) |
|
|
GlutenWholeTextFileV2Suite |
🔴 (1 failures) |
🔴 (1 failures) |
|
|
execution.python
| Suite |
Spark 4.0 |
Spark 4.1 |
Owner |
Comments |
GlutenPythonDataSourceSuite |
🔴 (1 failures) |
�� |
|
|
GlutenPythonUDFSuite |
🔴 (1 failures) |
🔴 |
|
|
GlutenPythonUDTFSuite |
🔴 |
🔴 |
|
|
GlutenRowQueueSuite |
🟢 |
🔴 |
|
Disabled in Spark 4.1 |
sources
| Suite |
Spark 4.0 |
Spark 4.1 |
Owner |
Comments |
GlutenBucketedReadWithHiveSupportSuite |
🔴 (2 failures) |
🔴 (2 failures) |
@Surbhi-Vijay |
|
GlutenBucketedWriteWithHiveSupportSuite |
🔴 (1 failures) |
🔴 (2 failures) |
|
Failure count changed: 1 → 2 |
GlutenCommitFailureTestRelationSuite |
🔴 (2 failures) |
🔴 (2 failures) |
|
|
GlutenDisableUnnecessaryBucketedScanWithHiveSupportSuite |
🔴 (2 failures) |
🔴 (2 failures) |
@Surbhi-Vijay |
|
GlutenJsonHadoopFsRelationSuite |
🔴 (2 failures) |
🔴 (2 failures) |
|
|
GlutenParquetHadoopFsRelationSuite |
🔴 (2 failures) |
🔴 (2 failures) |
|
|
GlutenSimpleTextHadoopFsRelationSuite |
🔴 (2 failures) |
🔴 (2 failures) |
|
|
streaming
Most of failures are same, it seems should be fixed in spark
| Suite |
Spark 4.0 |
Spark 4.1 |
Owner |
Comments |
GlutenEventTimeWatermarkSuite |
🔴 |
🔴 |
|
|
GlutenFileStreamSourceSuite |
🔴 |
🔴 |
|
|
GlutenFlatMapGroupsWithStateDistributionSuite |
🔴 |
🔴 |
|
|
GlutenFlatMapGroupsWithStateSuite |
🔴 |
🔴 |
|
|
GlutenRocksDBStateStoreFlatMapGroupsWithStateSuite |
🔴 |
🔴 |
|
|
GlutenRocksDBStateStoreStreamingAggregationSuite |
🔴 |
🔴 |
|
|
GlutenRocksDBStateStoreStreamingDeduplicationSuite |
🔴 |
🔴 |
|
|
GlutenStreamSuite |
🔴 |
🔴 |
|
|
GlutenStreamingAggregationDistributionSuite |
🔴 |
🔴 |
|
|
GlutenStreamingAggregationSuite |
🔴 |
🔴 |
|
|
GlutenStreamingDeduplicationDistributionSuite |
🔴 |
🔴 |
|
|
GlutenStreamingDeduplicationSuite |
🔴 |
🔴 |
|
|
GlutenStreamingInnerJoinSuite |
🔴 |
🔴 |
|
|
GlutenStreamingOuterJoinSuite |
🔴 |
🔴 |
|
|
GlutenStreamingSessionWindowDistributionSuite |
🔴 |
🔴 |
|
|
GlutenStreamingStateStoreFormatCompatibilitySuite |
🔴 |
🔴 |
|
|
GlutenTransformWithStateClusterSuite |
🔴 (spark.test.home) |
🔴 (spark.test.home) |
|
Discovered in #11800 |
Summary Statistics
- Total suites disabled in Spark 4.0: 70
- Total suites disabled in Spark 4.1: 76
- Unique suites across both versions: 76
This issue was updated based on #11800.
