Skip to content

Rebase vendored Substrait proto from v0.23.0 to 0.98.0 #12597

Description

@nielspardon

Background

Gluten vendors its Substrait .proto files (gluten-substrait/src/main/resources/substrait/proto/substrait/) pinned near Substrait v0.23.0 with local patches, while upstream Substrait is now at 0.98.0. The JVM plan producer and both native consumers (Velox, ClickHouse) regenerate from this single proto source and ship together, and Gluten's Substrait plans are transient (built per query, serialized across JNI, discarded) — so there is no wire-compatibility constraint and the rebase is a pure code migration.

Goal

Bring the vendored proto to official 0.98.0 core (verbatim) plus Gluten's genuinely engine-specific additions grafted at non-colliding numbers, migrating the JVM producer and both native backends in lockstep.

Approach

Land as small, independently-green PRs, one per message family (each keeps protoc + JVM + both native builds compiling). Minimal rebase first — relocate only colliding fork fields and drop dead/decided-drop items; extracting engine-specific forks to formal Substrait extensions is deferred to a follow-up.

Increments

  • 0 — Remove unused vendored proto files + dead proto-based type derivation + deprecated Expression.Enum
  • 1plan.proto / extensions.proto → 0.98 (URI→URN, AdvancedExtension.optimization repeated)
  • 2 — Type system: temporal types → PrecisionTimestamp*; drop Type.Nothing
  • 3CrossRelNestedLoopJoinRel (+ join-type enum remap)
  • 4 — Windowing: WindowRelConsistentPartitionWindowRel; WindowTypeBoundsType
  • 5FetchRel / AggregateRel.Grouping / ExpandRel
  • 6TopNRel / WriteRel
  • 7ReadRel (relocate stream_kafka; text-options redesign; VirtualTable)
  • 8Rel oneof reconciliation to exact 0.98 layout

Related: upstream consolidation work is tracked under the apache-gluten label in substrait-io/substrait.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions