Description
Add native Iceberg format-version 3 deletion-vector support to the Velox backend.
Today Gluten falls back when reading Puffin deletion vectors and row-level DELETE, UPDATE, and MERGE operations do not use the enhanced native Iceberg writer. Velox already contains the required deletion-vector writer support.
Proposed scope:
- Pass Puffin deletion-vector metadata through Substrait and map it to Velox
FileContent::kDeletionVector.
- Validate deletion vectors per delete file while continuing to fall back for equality deletes and malformed metadata.
- Offload Iceberg V3
WriteDeltaExec operations for DELETE, UPDATE, and MERGE when enhanced native writes are enabled.
- Preserve Iceberg's existing driver-side commit, validation, conflict detection, and abort handling.
- Seed native writes with existing deletion vectors so repeated mutations union deleted positions and replace only the prior active DV.
- Keep Spark 3.3 and Iceberg V2 behavior unchanged.
- Cover Spark 3.4, 3.5, 4.0, and 4.1 compatibility, native writer behavior, partitioned tables, and split-boundary reads.
Gluten version
main branch
Description
Add native Iceberg format-version 3 deletion-vector support to the Velox backend.
Today Gluten falls back when reading Puffin deletion vectors and row-level DELETE, UPDATE, and MERGE operations do not use the enhanced native Iceberg writer. Velox already contains the required deletion-vector writer support.
Proposed scope:
FileContent::kDeletionVector.WriteDeltaExecoperations for DELETE, UPDATE, and MERGE when enhanced native writes are enabled.Gluten version
main branch