Skip to content

[VL] Support Iceberg V3 deletion vectors in Velox #12628

Description

@wirybeaver

Description

Add native Iceberg format-version 3 deletion-vector support to the Velox backend.

Today Gluten falls back when reading Puffin deletion vectors and row-level DELETE, UPDATE, and MERGE operations do not use the enhanced native Iceberg writer. Velox already contains the required deletion-vector writer support.

Proposed scope:

  • Pass Puffin deletion-vector metadata through Substrait and map it to Velox FileContent::kDeletionVector.
  • Validate deletion vectors per delete file while continuing to fall back for equality deletes and malformed metadata.
  • Offload Iceberg V3 WriteDeltaExec operations for DELETE, UPDATE, and MERGE when enhanced native writes are enabled.
  • Preserve Iceberg's existing driver-side commit, validation, conflict detection, and abort handling.
  • Seed native writes with existing deletion vectors so repeated mutations union deleted positions and replace only the prior active DV.
  • Keep Spark 3.3 and Iceberg V2 behavior unchanged.
  • Cover Spark 3.4, 3.5, 4.0, and 4.1 compatibility, native writer behavior, partitioned tables, and split-boundary reads.

Gluten version

main branch

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions