You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The native ORC source was removed (see the commit referencing this issue) because its scan engine,
datafusion-orc, lags DataFusion releases: it has no published version (or upstream rev) for the
DataFusion the engine tracks, so every DataFusion bump forced a personally-maintained fork pin —
exactly the machine-specific/maintainer-specific coupling the project avoids. ORC tables now fall
back to Flink's own ORC format like any other unmatched source.
What removal took out, i.e. what restoring needs to bring back:
Native: the datafusion-orc + orc-rust dependencies, the openOrc JNI entry point and
ORC schema probe in the file-scan module (the split-ranged FileScan core is shared with Parquet
and still exists — ORC was only a different FileSource constructed into it).
Planner: the ORC source matcher and its physical/exec nodes, plus the orcSource toggle wiring.
Operator: the ORC bulk format subclass of the shared native-file bulk format.
Tests: the orc-baseline Maven module holding the host-parity test — isolated because Flink's
ORC format drags in orc-core 1.5.6 (protobuf-java 2.5), which cannot share a classpath with the
protobuf 4.x the main module's protobuf tests need.
Preconditions for restoring:
datafusion-orc publishes (or at least merges on main) support for the DataFusion version we track — chore: update datafusion to 54.0.0 datafusion-orc#151 is the DataFusion 54 bump we carried;
arrow 59 / DataFusion 55 will need the same treatment.
Failing that, a maintained ORC path via orc-rust's own reader driven by the shared FileScan is an
alternative that drops the datafusion-orc dependency entirely.
The native ORC source was removed (see the commit referencing this issue) because its scan engine,
datafusion-orc, lags DataFusion releases: it has no published version (or upstream rev) for the
DataFusion the engine tracks, so every DataFusion bump forced a personally-maintained fork pin —
exactly the machine-specific/maintainer-specific coupling the project avoids. ORC tables now fall
back to Flink's own ORC format like any other unmatched source.
What removal took out, i.e. what restoring needs to bring back:
datafusion-orc+orc-rustdependencies, theopenOrcJNI entry point andORC schema probe in the file-scan module (the split-ranged
FileScancore is shared with Parquetand still exists — ORC was only a different
FileSourceconstructed into it).orcSourcetoggle wiring.orc-baselineMaven module holding the host-parity test — isolated because Flink'sORC format drags in orc-core 1.5.6 (protobuf-java 2.5), which cannot share a classpath with the
protobuf 4.x the main module's protobuf tests need.
Preconditions for restoring:
chore: update datafusion to 54.0.0 datafusion-orc#151 is the DataFusion 54 bump we carried;
arrow 59 / DataFusion 55 will need the same treatment.
alternative that drops the datafusion-orc dependency entirely.