[VL][DELTA] Support Delta CDF scan offload by malinjawi · Pull Request #12218 · apache/gluten

malinjawi · 2026-06-01T14:12:32Z

What changes are proposed in this pull request?

Addresses #12195.

Delta CDF reads enter Spark as CDCReader.DeltaCDFRelation, so they do not initially have the normal FileSourceScanExec + DeltaParquetFileFormat shape that Gluten's existing Delta scan offload rule recognizes.

This PR adds a Gluten Delta planner strategy, wired from the Velox Delta component, that recognizes DeltaCDFRelation, expands it through Delta's own CDF batch planning path, and rewrites the original projection/filter attributes onto the expanded logical plan. After that, the existing Delta scan offload path can plan the underlying CDF file scans as DeltaScanTransformer.

The change is intentionally scoped to Gluten's Delta/Spark planning layer rather than Velox C++:

Add DeltaCDFScanStrategy for table_changes(...) and DataFrame readChangeFeed scans.
Add Delta-version helper shims for Delta 2.3, 2.4, 3.3, and 4.x API differences.
Register the planner strategy from VeloxDeltaComponent.
Add Delta regression coverage for insert/update/delete CDF rows, filter/projection handling, bounded version reads, DataFrame readChangeFeed, column mapping, and a startingVersion = 0 case.

One planner/test nuance: Delta CDF expansion can keep a Spark-side ExistingRDD branch for synthesized change rows, including on the tested update/delete CDF paths. The regression suite therefore compares results against vanilla Spark and asserts that the expanded CDF file scans are transformed to DeltaScanTransformer, rather than requiring the entire expanded CDF union to be globally fallback-free.

How was this patch tested?

Local checks used JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home.

git diff --check
./dev/format-scala-code.sh check
./build/mvn -pl gluten-delta -am -Pspark-3.5 -Pjava-17 -Pbackends-velox -Pdelta -DskipTests test-compile
./build/mvn -pl gluten-delta -am -Pspark-4.0 -Pscala-2.13 -Pjava-17 -Pbackends-velox -Pdelta -DskipTests test-compile
./build/mvn -pl gluten-delta -am -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pbackends-velox -Pdelta -DskipTests test-compile

Earlier cross-version compile checks also passed before the test-expectation-only update:

./build/mvn -pl gluten-delta -am -Pspark-3.3 -Pjava-17 -Pbackends-velox -Pdelta -DskipTests -Dcheckstyle.skip=true -Dscalastyle.skip=true -Dspotless.check.skip=true test-compile
./build/mvn -pl gluten-delta -am -Pspark-3.4 -Pjava-17 -Pbackends-velox -Pdelta -DskipTests -Dcheckstyle.skip=true -Dscalastyle.skip=true -Dspotless.check.skip=true test-compile
./build/mvn -pl gluten-delta -am -Pspark-3.5 -Pjava-17 -Pbackends-velox -Pdelta -DskipTests -Dcheckstyle.skip=true -Dscalastyle.skip=true -Dspotless.check.skip=true test-compile
./build/mvn -pl gluten-delta -am -Pspark-4.0 -Pscala-2.13 -Pjava-17 -Pbackends-velox -Pdelta -DskipTests -Dcheckstyle.skip=true -Dscalastyle.skip=true -Dspotless.check.skip=true test-compile
./build/mvn -pl gluten-delta -am -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pbackends-velox -Pdelta -DskipTests -Dcheckstyle.skip=true -Dscalastyle.skip=true -Dspotless.check.skip=true test-compile

Full native Velox runtime and benchmarking are left to CI / a native Gluten-optimized environment; this local checkout does not have cpp/build/releases/libgluten.so and the Velox external project build available.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: IBM BOB

github-actions · 2026-06-01T14:13:06Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-06-01T14:59:43Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-06-01T15:08:37Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-06-01T15:52:58Z

Run Gluten Clickhouse CI on x86

felipepessoto · 2026-06-02T19:53:56Z

Could you add a DV-enabled CDF regression test to confirm the behavior? Something like:

  CREATE TABLE t (id INT, name STRING) USING delta
  TBLPROPERTIES ('delta.enableChangeDataFeed'='true', 'delta.enableDeletionVectors'='true');
  INSERT INTO t VALUES (1,'v1'),(2,'v2'),(3,'v3');
  DELETE FROM t WHERE id = 2;
  SELECT id, name, _change_type FROM table_changes('t', 0);

compared against vanilla Spark (checkAnswer)

felipepessoto · 2026-06-02T19:55:27Z

+import org.apache.spark.sql.delta.BatchCDFSchemaEndVersion
+import org.apache.spark.sql.delta.commands.cdc.CDCReader
+
+object DeltaCDFRelationHelper {


Do you know what happens in Delta 4.1/4.2/4.3? Would it take the code from 4.0 folder?

Yes @felipepessoto I checked the current Maven profile/source wiring:

spark-4.0 sets delta.version=4.0.1 and delta.binary.version=40

spark-4.1 sets delta.version=4.1.0 and delta.binary.version=40

the Delta profile adds sources from src-delta${delta.binary.version}/main/scala

So with the current profile setup, both Spark 4.0 / Delta 4.0.x and Spark 4.1 / Delta 4.1.x use gluten-delta/src-delta40/....

For future Delta 4.2 / 4.3 support, it depends on how those profiles are introduced. If they keep delta.binary.version=40, they will continue to use the same src-delta40 helper. If Delta changes the relevant CDF APIs and Gluten introduces a new binary bucket, then we should add a new version-specific helper.

This also relates to the earlier folder-activation discussion in #11924: supporting family-level folders like src-spark4 / src-delta4 would make this cleaner and avoid copying code across Spark/Delta 4.x profiles when the APIs stay compatible. I think that belongs in a separate Maven/source-layout refactor rather than in this CDF PR.

malinjawi · 2026-06-08T08:26:26Z

Could you add a DV-enabled CDF regression test to confirm the behavior? Something like:
  CREATE TABLE t (id INT, name STRING) USING delta
  TBLPROPERTIES ('delta.enableChangeDataFeed'='true', 'delta.enableDeletionVectors'='true');
  INSERT INTO t VALUES (1,'v1'),(2,'v2'),(3,'v3');
  DELETE FROM t WHERE id = 2;
  SELECT id, name, _change_type FROM table_changes('t', 0);
compared against vanilla Spark (checkAnswer)

Thanks @felipepessoto good point! I agree this is worth covering explicitly.

I’ll add a focused DV-enabled CDF regression with delta.enableChangeDataFeed=true and delta.enableDeletionVectors=true, comparing table_changes(...) against vanilla Spark via the existing runAndCompare / checkAnswer path.

One important nuance: this PR is not intended to claim full native DV support. DV-backed Delta scan execution is still a separate active area in Gluten/Velox and may continue to fall back where native DV handling is not available. The goal of the regression here is to make sure enabling DV does not break the CDF planning path or result correctness.

github-actions · 2026-06-08T08:46:33Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-06-08T11:13:19Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-06-16T09:51:26Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-06-16T16:35:53Z

Run Gluten Clickhouse CI on x86

felipepessoto · 2026-06-16T16:44:04Z

+  // DeltaScanTransformer).
+  private def touchesDeletionVectors(cdfRelation: CDCReader.DeltaCDFRelation): Boolean =
+    cdfRelation.snapshotWithSchemaMode.snapshot.metadata.configuration
+      .get("delta.enableDeletionVectors")


This table property says if new writes can write DV. But I think you want to find if table has any DV

Good catch; you were right that the table property only controls future writes. This is now addressed in two layers:

3b462034a makes DeltaCDFScanStrategy inspect AddFile/RemoveFile actions only in the requested CDF version range.

Spark 3.4 CI then exposed an older physical fallback that still treated a DV-capable TahoeFileIndex as if every CDF range contained a DV. 049e608a3 now special-cases CdcAddFileIndex and TahoeRemoveFileIndex there as well, using their exact files/actions before the generic table-level fallback.

The regression covers both directions: a DV-enabled table's insert-only 0..1 range must offload, while a range containing a DV-backed delete must fall back even after delta.enableDeletionVectors is set back to false, with the CDF rows checked for correctness. Formatting and test compilation pass for Spark 3.3, 3.4, 3.5, 4.0, and 4.1.

Delta change-data-feed reads enter Spark as a CDCReader.DeltaCDFRelation, which does not have the FileSourceScanExec + DeltaParquetFileFormat shape that Gluten's Delta scan offload recognizes. Add a Gluten Delta planner strategy (wired from VeloxDeltaComponent) that recognizes DeltaCDFRelation, expands it through Delta's own CDF batch planning, and rewrites the projection/filter attributes onto the expanded plan, so the existing Delta scan offload path can plan the underlying CDF file scans as DeltaScanTransformer. CDF over a deletion-vector-enabled table is kept on Spark: native CDF lacks the DV-aware row-level reconciliation, so an offloaded CDF scan would emit still-live rows as `delete` change rows. Re-planning Delta's analyzed CDF batch plan also drops that reconciliation on some Delta versions (Delta 2.4 / Spark 3.4), where the remove side would surface every row of a logically removed file as a `delete` change row. The planner strategy therefore declines to intercept CDF reads whose table has deletion vectors enabled and leaves them to Delta's own DeltaCDFRelation scan, which reconciles DVs correctly on every supported Delta version; DeltaScanTransformer additionally guards both CDF scan sides -- the add side (CdcAddFileIndex) and the remove side (TahoeRemoveFileIndex) -- as a backstop. Normal (non-CDF) DV scans are unaffected and continue to apply the DV natively. Addresses apache#12195.

github-actions · 2026-06-16T19:36:45Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-07-19T09:39:38Z

Run Gluten Clickhouse CI on x86

Inspect AddFile and RemoveFile actions in the requested CDF version range instead of relying on the table property, which only controls future DV writes.\n\nCover both DV-enabled ranges without DV actions and existing DV actions after the property is disabled.

github-actions · 2026-07-19T10:49:09Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-07-19T12:01:16Z

Run Gluten Clickhouse CI on x86

felipepessoto · 2026-07-20T19:19:37Z

  }

  override def injectRules(injector: Injector): Unit = {
+    injector.spark.injectPlannerStrategy(DeltaCDFScanStrategy(_))


In general, do we create individual flags for each feature we offload? If so, could we have one for CDF offload?

Yes, good suggestion. I added a dedicated experimental guard in 6517820:

spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan

It defaults to true, is checked against the active session config on each planning pass, and setting it to false makes DeltaCDFScanStrategy decline the relation so Delta uses its vanilla CDF path. I also added a regression test for the disabled path and documented the setting.

felipepessoto · 2026-07-20T23:27:26Z

@malinjawi I have run the Delta CI with your changes, there are only a few failures related to filter push down, I guess it is not a blocker, but could you please double check?

https://github.com/apache/gluten/actions/runs/29772714951/job/88458293204?pr=12582
https://github.com/apache/gluten/actions/runs/29772714951/job/88458293208?pr=12582
https://github.com/apache/gluten/actions/runs/29772714951/job/88458293260?pr=12582
https://github.com/apache/gluten/actions/runs/29772714951/job/88458293225?pr=12582

malinjawi · 2026-07-24T12:21:33Z

Thanks @felipepessoto, I double-checked the combined Delta CI. The 13 reported regressions all had the same root cause: DeltaCDFScanStrategy expands the CDF relation during physical planning, after Spark's normal optimizer pass, so the reconstructed filter stayed above the CDF union and never reached the Delta file scans. Row correctness was unaffected, but this was a real predicate-pushdown regression rather than a baseline issue.

I fixed it in d740b0d by optimizing the resolved replacement plan before handing it back to physical planning. I also added a regression assertion that verifies the id predicate is present in DeltaScanTransformer.dataFilters. The change passes Spotless and clean test-compiles for Spark 3.3, 3.4, 3.5, 4.0, and 4.1.

Could you please rerun the combined Delta CI against the updated head? It should specifically clear the filter-pushdown failures from run 29772714951.

github-actions · 2026-07-24T12:34:11Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-07-24T13:01:53Z

Run Gluten Clickhouse CI on x86

malinjawi · 2026-07-24T13:17:17Z

Update: the standard x86 Spark 3.3 job caught a separate output-attribute regression in the first pushdown fix. Optimizing the whole reconstructed plan allowed Spark 3.3 to remove the aliases that preserve the original CDF relation expression IDs, so a parent sort later tried to bind a stale name attribute.

I fixed that in bf2ac70 by optimizing only the filtered CDF child, then applying the output-restoring Project after the optimizer. This keeps predicate pushdown while preserving the expression IDs expected by already-planned parent operators. Spotless and Delta test-compiles pass for Spark 3.3 and Spark 4.1, and a fresh standard CI run is now queued on this head.

The combined Delta run 29772714951 is still the earlier run against the old combined branch. @felipepessoto, could you please rerun it against bf2ac70 once the branch is refreshed?

github-actions · 2026-07-24T13:54:01Z

Run Gluten Clickhouse CI on x86

felipepessoto · 2026-07-24T15:18:29Z

@malinjawi I synced your 3 new commits and started CI here: #12582

felipepessoto · 2026-07-24T22:14:42Z

@malinjawi I can see the Pushed Filters, but there is a difference, the expected filter contains a *, do you know what this mean?

...PushedFilters: [IsNotNull(`id with space`), LessThan(`id with space`,5)], ReadSchema: struct<col-215ff29d-abc0-48ea-92fc-0c09b73e9f40:bigint,col-c6bb9720-4fc4-4181-b423-bf2050fbb9dc:b... NativeFilters: [isnotnull(col-c6bb9720-4fc4-4181-b423-bf2050fbb9dc#1159077L),(col-c6bb9720-4fc4-4181-b423-bf2050fbb9dc#1159077L < 5)]
[info]   )" did not contain "PushedFilters: [*IsNotNull(id with space), *LessThan(id with space,5)]" (DeltaCDCColumnMappingSuite.scala:671)

I asked AI to explain, but I'm not familiar with this and I can't confirm, I hope you know more:

What the * means

In a scan's PushedFilters metadata, a * prefix marks a filter the data source guarantees to fully evaluate itself, so Spark drops its own redundant Filter above the scan. It's emitted only by RowDataSourceScanExec ( DataSourceScanExec.scala:168 ):

if (handledFilters.contains(filter)) s"*$filter" else s"$filter"

handledFilters = pushed filters minus the source's unhandledFilters(...) . A filter without * is offered to the source but Spark still re-applies it (best-effort, for skipping only).

Why the offloaded plan has no *

It's a different scan node, not a "less pushed-down" one.

Before the PR — readChangeFeed produces DeltaCDFRelation , which is BaseRelation with CatalystScan and declares unhandledFilters = Array.empty ( CDCReaderBase.scala:109 ). Spark plans it as a RowDataSourceScanExec ; since nothing is unhandled, every filter is "handled" and gets a * :

PushedFilters: [*IsNotNull(id), *LessThan(id,5)] ✅ test matches

After the PR — DeltaCDFScanStrategy expands that relation into the underlying Parquet plan, which Gluten offloads to DeltaScanTransformer (a FileSourceScanLike ). Its metadata comes from FileSourceScanLike.metadata ( DataSourceScanExec.scala:444 ), which never marks filters:

"PushedFilters" -> seqToString(pushedFiltersForDisplay) // no "*"

PushedFilters: [IsNotNull(id), LessThan(id,5)] ... NativeFilters: [...] ❌ no "*"

So the exact substring [*IsNotNull(id), *LessThan(id,5)] disappears and the assertion fails.

The key point: the filter is still pushed down — it's in DeltaScanTransformer.dataFilters (your own test asserts exactly that). File-source scans (Parquet, hence Velox) never emit * , because Parquet pushdown is best-effort row-group skipping and Spark always keeps a post-scan filter for correctness. The * is a CatalystScan / PrunedFilteredScan "I own this filter" marker that a file scan structurally cannot reproduce.

https://github.com/apache/gluten/actions/runs/30104513088/job/89521902732?pr=12582
https://github.com/apache/gluten/actions/runs/30104513088/job/89521902757?pr=12582
https://github.com/apache/gluten/actions/runs/30104513088/job/89521902748?pr=12582
https://github.com/apache/gluten/actions/runs/30104513088/job/89521902791?pr=12582

github-actions Bot added VELOX DATA_LAKE labels Jun 1, 2026

malinjawi marked this pull request as ready for review June 1, 2026 18:50

felipepessoto reviewed Jun 2, 2026

View reviewed changes

malinjawi force-pushed the codex/delta-cdf-offload branch from 2cd31c0 to bca92ba Compare June 16, 2026 09:46

malinjawi force-pushed the codex/delta-cdf-offload branch from bca92ba to b14394d Compare June 16, 2026 16:30

felipepessoto reviewed Jun 16, 2026

View reviewed changes

malinjawi force-pushed the codex/delta-cdf-offload branch from b14394d to 582641a Compare June 16, 2026 19:36

Merge branch 'main' into codex/delta-cdf-offload

e0d9027

[VL][Delta] Check CDF indexes for deletion vectors

049e608

malinjawi requested a review from felipepessoto July 19, 2026 18:00

felipepessoto reviewed Jul 20, 2026

View reviewed changes

felipepessoto approved these changes Jul 20, 2026

View reviewed changes

[VL][Delta] Preserve CDF filter pushdown

d740b0d

[VL][Delta] Add CDF offload guard

6517820

github-actions Bot added the DOCS label Jul 24, 2026

[VL][Delta] Preserve CDF output attributes

bf2ac70

Uh oh!

Conversation

malinjawi commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes are proposed in this pull request?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

github-actions Bot commented Jun 1, 2026

Uh oh!

github-actions Bot commented Jun 1, 2026

Uh oh!

github-actions Bot commented Jun 1, 2026

Uh oh!

github-actions Bot commented Jun 1, 2026

Uh oh!

felipepessoto commented Jun 2, 2026

Uh oh!

felipepessoto Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

malinjawi Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

malinjawi commented Jun 8, 2026

Uh oh!

github-actions Bot commented Jun 8, 2026

Uh oh!

github-actions Bot commented Jun 8, 2026

Uh oh!

github-actions Bot commented Jun 16, 2026

Uh oh!

github-actions Bot commented Jun 16, 2026

Uh oh!

felipepessoto Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

malinjawi Jul 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 16, 2026

Uh oh!

github-actions Bot commented Jul 19, 2026

Uh oh!

github-actions Bot commented Jul 19, 2026

Uh oh!

github-actions Bot commented Jul 19, 2026

Uh oh!

felipepessoto Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

malinjawi Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

felipepessoto commented Jul 20, 2026

Uh oh!

malinjawi commented Jul 24, 2026

Uh oh!

github-actions Bot commented Jul 24, 2026

Uh oh!

github-actions Bot commented Jul 24, 2026

Uh oh!

malinjawi commented Jul 24, 2026

Uh oh!

github-actions Bot commented Jul 24, 2026

Uh oh!

felipepessoto commented Jul 24, 2026

Uh oh!

felipepessoto commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

malinjawi commented Jun 1, 2026 •

edited

Loading

malinjawi Jul 19, 2026 •

edited

Loading

felipepessoto commented Jul 24, 2026 •

edited

Loading