Columnar endpoints beyond local Parquet: Iceberg, and remote filesystems for the source

[jordepic]

What

The native Parquet source is local-only: its matcher falls back on any non-file: scheme. The columnar source→sink win is among the project's strongest results, but real pipelines read Iceberg tables and remote filesystems (HDFS, S3/object store). Extending the columnar read endpoints there is what makes the win apply to actual workloads.

Update 2026-07-07 — the sink half is done: the Parquet sink now writes to any scheme Flink has a filesystem for (the native side encodes bytes only; Flink's own recoverable streams do the IO and exactly-once commit), with full PARTITIONED BY support. See divergences/22 and docs/coverage-and-fallbacks.md §5. Remaining sink tails are tracked in #20, and a from-first-principles native-IO sink (object_store multipart) in #21.

Remaining scope

• Remote filesystems for the Parquet source: hdfs:/s3: paths. The native reader uses local std::fs; remote needs an object-store/HDFS reader (DataFusion's object_store is already threaded through the scan config as ObjectStoreUrl::local_filesystem() — a registered remote store slots in there). The matcher's scheme gate is where support plugs in. Unlike the sink, the source cannot lean on Flink for the IO: the bytes must reach the native decoder.
• Iceberg source/sink: catalog resolution, split planning, schema evolution, and Iceberg's metadata/commit protocol. Much larger than a filesystem path. The native side can read/write Parquet data files as Arrow; the Iceberg table semantics (snapshots, manifests, commits) are the work. Consider iceberg-rust on the native side.

Parity / correctness

• Each endpoint must produce/consume byte-identical data to Flink's equivalent connector, verified by the harness (write via both, read both back; or read via both, compare).
• Exactly-once commit semantics must match the connector being replaced (Iceberg snapshot commit).