[GLUTEN-11550][VL][UT] Enable Variant test suites by baibaichen · Pull Request #11726 · apache/gluten

baibaichen · 2026-03-09T08:59:52Z

What changes are proposed in this pull request?

Enable GlutenVariantEndToEndSuite, GlutenVariantShreddingSuite, and GlutenParquetVariantShreddingSuite for both spark40 and spark41.

Changes:

VeloxValidatorApi.scala: Detect variant shredded structs (produced by Spark's PushVariantIntoScan) by checking for __VARIANT_METADATA_KEY metadata on struct fields. Triggers fallback to Spark's native Parquet reader since Velox cannot read variant shredding encoding.
ParquetMetadataUtils.scala: Add variant annotation check in isUnsupportedMetadata, gated by parquetMetadataValidationEnabled. Reads the same footer, no extra I/O.
Spark41Shims.scala: Add shouldFallbackForParquetVariantAnnotation to detect Parquet variant logical type annotations.
GlutenParquetVariantShreddingSuite (spark41): Set parquetMetadataValidationEnabled=true to enable variant annotation fallback detection.
pom.xml: Add -Dfile.encoding=UTF-8 to test JVM args. On JDK 17 with LANG=C (CI containers centos-8/9), the default charset is US-ASCII causing garbled output for multi-byte characters. JDK 18+ defaults to UTF-8 via JEP 400. See VariantUtil.java#L508.

How was this patch tested?

spark40: GlutenVariantEndToEndSuite 14✅, GlutenVariantShreddingSuite 8✅, GlutenParquetVariantShreddingSuite 5✅
spark41: GlutenVariantEndToEndSuite 14✅, GlutenVariantShreddingSuite 8✅, GlutenParquetVariantShreddingSuite 7✅

Was this patch authored or co-authored using generative AI tooling?

Generated-by: GitHub Copilot CLI

Related issue: #11550
Note: This PR subsumes #11723 (GlutenParquetVariantShreddingSuite).

github-actions · 2026-03-11T05:14:24Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-03-11T09:53:34Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-03-11T16:23:27Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-03-13T15:01:48Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-03-16T02:37:31Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-03-16T07:19:16Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-03-16T07:20:45Z

Run Gluten Clickhouse CI on x86

baibaichen · 2026-03-16T09:13:48Z

@jinchengchenghh @zhztheplayer please review

Copilot

Pull request overview

Adds Spark 4.1-specific detection and fallback behavior for Parquet Variant annotation / variant shredding so Velox avoids reading unsupported encodings, and enables previously-disabled Variant-related test suites.

Changes:

Add Spark shim hooks to decide when Parquet Variant annotation should trigger fallback (Spark 4.1).
Refactor Parquet metadata validation to read each footer once and include an always-on Variant annotation check when needed.
Enable Variant-related test suites for Spark 4.0/4.1 and force UTF-8 file encoding for test JVMs.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
shims/spark41/src/main/scala/org/apache/gluten/sql/shims/spark41/Spark41Shims.scala	Implements Spark 4.1 logic for checking Parquet Variant annotations via footer schema.
shims/common/src/main/scala/org/apache/gluten/sql/shims/SparkShims.scala	Adds default shim APIs for Variant-annotation fallback decisions.
backends-velox/src/main/scala/org/apache/gluten/utils/ParquetMetadataUtils.scala	Refactors footer scanning and adds Variant-annotation-driven fallback path.
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxValidatorApi.scala	Detects “variant shredded” StructType via field metadata and rejects it for Velox.
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxBackend.scala	Minor cleanup around metadata validation call chaining.
gluten-ut/spark41/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala	Enables Variant-related suites for Spark 4.1.
gluten-ut/spark40/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala	Enables Variant-related suites for Spark 4.0.
pom.xml	Forces UTF-8 file.encoding for test JVM execution.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

backends-velox/src/main/scala/org/apache/gluten/utils/ParquetMetadataUtils.scala

backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxValidatorApi.scala

jinchengchenghh · 2026-03-16T11:12:23Z

backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxValidatorApi.scala

+        // These structs have all fields annotated with __VARIANT_METADATA_KEY metadata.
+        // Velox cannot read the variant shredding encoding in Parquet files.
+        if (
+          struct.fields.nonEmpty &&


Do we need to add nonEmpty check?

Yes, nonEmpty is needed because forall returns true for an empty collection. Without it, an empty struct would be incorrectly detected as variant shredded.

jinchengchenghh · 2026-03-16T11:13:15Z

backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxValidatorApi.scala

+          struct.fields.nonEmpty &&
+          struct.fields.forall(_.metadata.contains("__VARIANT_METADATA_KEY"))
+        ) {
+          return Some(s"Variant shredded struct is not supported: $struct")


Might the struct $struct too heavy?

This is consistent with the existing pattern at line 144: s"Schema / data type not supported: $schema" which also prints the full type. Since this is a validation failure message (logged once per scan fallback, not per row), the overhead is negligible.

jinchengchenghh · 2026-03-18T10:03:57Z

backends-velox/src/main/scala/org/apache/gluten/utils/ParquetMetadataUtils.scala

-    if (!GlutenConfig.get.parquetMetadataValidationEnabled) {
-      None
+    val enabled = GlutenConfig.get.parquetMetadataValidationEnabled
+    if (enabled || SparkShimLoader.getSparkShims.needsVariantAnnotationCheck) {


Can we reuse parquetMetadataValidationEnabled? the default value is false

Updated. Variant annotation check is now gated by parquetMetadataValidationEnabled (same as other checks). For Spark 4.1 GlutenParquetVariantShreddingSuite, we set parquetMetadataValidationEnabled=true via sparkConf in the test suite.

jinchengchenghh · 2026-03-18T10:07:19Z

shims/common/src/main/scala/org/apache/gluten/sql/shims/SparkShims.scala


  def isParquetFileEncrypted(footer: ParquetMetadata): Boolean

+  def shouldFallbackForParquetVariantAnnotation(footer: ParquetMetadata): Boolean = false


We would we not fallback?

Default false because Spark 3.x and 4.0 don't have Parquet variant logical type annotations — only Spark 4.1 introduced VariantLogicalTypeAnnotation. Spark41Shims overrides this to actually check the Parquet schema. For older Spark versions, there's nothing to detect, so no fallback needed.

github-actions · 2026-03-20T09:37:09Z

Run Gluten Clickhouse CI on x86

Enable GlutenVariantEndToEndSuite, GlutenVariantShreddingSuite, and GlutenParquetVariantShreddingSuite for both spark40 and spark41. Changes: 1. VeloxValidatorApi: Detect variant shredded structs produced by Spark's PushVariantIntoScan (checking __VARIANT_METADATA_KEY metadata) to trigger fallback to Spark's native Parquet reader. 2. ParquetMetadataUtils: Add variant annotation check in isUnsupportedMetadata, gated by parquetMetadataValidationEnabled. Reads the same footer, no extra I/O. 3. Spark41Shims: Add shouldFallbackForParquetVariantAnnotation to detect Parquet variant logical type annotations. 4. GlutenParquetVariantShreddingSuite (spark41): Set parquetMetadataValidationEnabled=true to enable variant annotation fallback detection. 5. pom.xml: Add -Dfile.encoding=UTF-8 to test JVM args. On JDK 17 with LANG=C (CI containers centos-8/9), the default charset is US-ASCII causing garbled output for multi-byte characters. JDK 18+ defaults to UTF-8 via JEP 400. See: https://github.com/apache/spark/blob/v4.0.1/common/variant/src/main/java/org/apache/spark/types/variant/VariantUtil.java#L508 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-03-20T09:54:00Z

Run Gluten Clickhouse CI on x86

jinchengchenghh

Thanks!

github-actions bot added CORE works for Gluten Core VELOX labels Mar 9, 2026

baibaichen mentioned this pull request Mar 9, 2026

Spark 4.x: Tracking disabled test suites #11550

Open

baibaichen force-pushed the fix/GlutenVariantEndToEndSuite-GlutenVariantShreddingSuite branch 3 times, most recently from 4a21089 to 794a1fe Compare March 11, 2026 05:13

baibaichen changed the title ~~[GLUTEN-11550][VL][UT] Enable GlutenVariantEndToEndSuite and GlutenVariantShreddingSuite~~ [GLUTEN-11550][VL][UT] Enable Variant test suites Mar 11, 2026

baibaichen mentioned this pull request Mar 11, 2026

[GLUTEN-11550][UT] Enable GlutenParquetVariantShreddingSuite for Spark 4.1 #11723

Closed

baibaichen force-pushed the fix/GlutenVariantEndToEndSuite-GlutenVariantShreddingSuite branch from 794a1fe to c6759c0 Compare March 11, 2026 09:53

baibaichen force-pushed the fix/GlutenVariantEndToEndSuite-GlutenVariantShreddingSuite branch from c6759c0 to d53e74e Compare March 11, 2026 16:15

baibaichen force-pushed the fix/GlutenVariantEndToEndSuite-GlutenVariantShreddingSuite branch from d53e74e to ff31977 Compare March 13, 2026 14:57

baibaichen force-pushed the fix/GlutenVariantEndToEndSuite-GlutenVariantShreddingSuite branch from ff31977 to f70182f Compare March 16, 2026 02:37

baibaichen force-pushed the fix/GlutenVariantEndToEndSuite-GlutenVariantShreddingSuite branch from f70182f to cbba654 Compare March 16, 2026 07:18

baibaichen force-pushed the fix/GlutenVariantEndToEndSuite-GlutenVariantShreddingSuite branch from cbba654 to aea9428 Compare March 16, 2026 07:19

baibaichen requested review from Copilot, jinchengchenghh and zhztheplayer March 16, 2026 09:13

Copilot started reviewing on behalf of baibaichen March 16, 2026 09:15 View session

Copilot AI reviewed Mar 16, 2026

View reviewed changes

jinchengchenghh reviewed Mar 18, 2026

View reviewed changes

baibaichen force-pushed the fix/GlutenVariantEndToEndSuite-GlutenVariantShreddingSuite branch from aea9428 to 6086667 Compare March 20, 2026 09:36

baibaichen force-pushed the fix/GlutenVariantEndToEndSuite-GlutenVariantShreddingSuite branch from 6086667 to ce27cd8 Compare March 20, 2026 09:53

jinchengchenghh approved these changes Mar 20, 2026

View reviewed changes

baibaichen merged commit 7a835c9 into apache:main Mar 20, 2026
59 checks passed

baibaichen deleted the fix/GlutenVariantEndToEndSuite-GlutenVariantShreddingSuite branch March 20, 2026 12:14


		def isParquetFileEncrypted(footer: ParquetMetadata): Boolean

		def shouldFallbackForParquetVariantAnnotation(footer: ParquetMetadata): Boolean = false

Conversation

baibaichen commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes are proposed in this pull request?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

github-actions bot commented Mar 11, 2026

Uh oh!

github-actions bot commented Mar 11, 2026

Uh oh!

github-actions bot commented Mar 11, 2026

Uh oh!

github-actions bot commented Mar 13, 2026

Uh oh!

github-actions bot commented Mar 16, 2026

Uh oh!

github-actions bot commented Mar 16, 2026

Uh oh!

github-actions bot commented Mar 16, 2026

Uh oh!

baibaichen commented Mar 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 20, 2026

Uh oh!

github-actions bot commented Mar 20, 2026

Uh oh!

jinchengchenghh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

baibaichen commented Mar 9, 2026 •

edited

Loading