Skip to content

Restore the native ORC file source #19

Description

@jordepic

The native ORC source was removed (see the commit referencing this issue) because its scan engine,
datafusion-orc, lags DataFusion releases: it has no published version (or upstream rev) for the
DataFusion the engine tracks, so every DataFusion bump forced a personally-maintained fork pin —
exactly the machine-specific/maintainer-specific coupling the project avoids. ORC tables now fall
back to Flink's own ORC format like any other unmatched source.

What removal took out, i.e. what restoring needs to bring back:

  • Native: the datafusion-orc + orc-rust dependencies, the openOrc JNI entry point and
    ORC schema probe in the file-scan module (the split-ranged FileScan core is shared with Parquet
    and still exists — ORC was only a different FileSource constructed into it).
  • Planner: the ORC source matcher and its physical/exec nodes, plus the orcSource toggle wiring.
  • Operator: the ORC bulk format subclass of the shared native-file bulk format.
  • Tests: the orc-baseline Maven module holding the host-parity test — isolated because Flink's
    ORC format drags in orc-core 1.5.6 (protobuf-java 2.5), which cannot share a classpath with the
    protobuf 4.x the main module's protobuf tests need.

Preconditions for restoring:

  • datafusion-orc publishes (or at least merges on main) support for the DataFusion version we track —
    chore: update datafusion to 54.0.0 datafusion-orc#151 is the DataFusion 54 bump we carried;
    arrow 59 / DataFusion 55 will need the same treatment.
  • Failing that, a maintained ORC path via orc-rust's own reader driven by the shared FileScan is an
    alternative that drops the datafusion-orc dependency entirely.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions