The native Parquet sink (encoding-only native, Flink owns IO/commit — divergences/22) honors the filesystem+parquet config surface 1:1 or falls back with a recorded reason (docs/coverage-and-fallbacks.md §5). These are the remaining fallback causes worth removing, roughly by value:
- INT96 timestamps (the biggest: Flink's default timestamp encoding, so any timestamp-bearing table without
parquet.write.int64.timestamp=true falls back). arrow-rs's ArrowWriter cannot write INT96 — its schema converter never targets it and a forced descriptor panics at write_leaf — but the low-level layer can (Int96ColumnWriter supports write_batch, statistics, and dictionaries). Since the sink already restricts itself to flat schemas, def-levels are trivial (validity bitmap), so a hand-rolled SerializedFileWriter path for INT96-bearing tables is feasible. Value layout is 12 bytes LE: nanos-of-day (8) + Julian day (4), floor semantics like the INT64 path.
- Nested written columns (ARRAY/MAP/MULTISET/ROW): needs group-encoding parity verification against Flink's
ParquetSchemaConverter (3-level lists with element, key_value maps with non-null keys that throw on null at write, MULTISET as map-to-INT32-count) plus forced descriptors for nested decimals.
parquet.utc-timezone=false timestamps (also a default): Flink shifts values through the JVM's local timezone via TimestampData.toTimestamp(). A chrono-tz implementation behind allowIncompatible would follow the divergences/17 precedent (tzdb-skew edges are why it must be gated).auto-compaction=true: Flink swaps in a different DAG (CompactCoordinator/CompactOperator). Either reuse those operators verbatim around the native writer (the divergences/22 pattern) or decline forever and document.- Encoder memory accounting: each open bucket buffers an in-progress row group (up to
parquet.block.size) in native memory, currently untracked by ManagedMemoryBudget. Same order as the host's parquet-mr heap buffering, but partitioned tables multiply it by active buckets per subtask.
The strict INT96/utc-timezone gates were a deliberate decision: physical-encoding differences a reader can observe fall back rather than silently diverge.
The native Parquet sink (encoding-only native, Flink owns IO/commit — divergences/22) honors the filesystem+parquet config surface 1:1 or falls back with a recorded reason (docs/coverage-and-fallbacks.md §5). These are the remaining fallback causes worth removing, roughly by value:
parquet.write.int64.timestamp=truefalls back). arrow-rs'sArrowWritercannot write INT96 — its schema converter never targets it and a forced descriptor panics atwrite_leaf— but the low-level layer can (Int96ColumnWritersupportswrite_batch, statistics, and dictionaries). Since the sink already restricts itself to flat schemas, def-levels are trivial (validity bitmap), so a hand-rolledSerializedFileWriterpath for INT96-bearing tables is feasible. Value layout is 12 bytes LE: nanos-of-day (8) + Julian day (4), floor semantics like the INT64 path.ParquetSchemaConverter(3-level lists withelement,key_valuemaps with non-null keys that throw on null at write, MULTISET as map-to-INT32-count) plus forced descriptors for nested decimals.parquet.utc-timezone=falsetimestamps (also a default): Flink shifts values through the JVM's local timezone viaTimestampData.toTimestamp(). A chrono-tz implementation behindallowIncompatiblewould follow the divergences/17 precedent (tzdb-skew edges are why it must be gated).auto-compaction=true: Flink swaps in a different DAG (CompactCoordinator/CompactOperator). Either reuse those operators verbatim around the native writer (the divergences/22 pattern) or decline forever and document.parquet.block.size) in native memory, currently untracked by ManagedMemoryBudget. Same order as the host's parquet-mr heap buffering, but partitioned tables multiply it by active buckets per subtask.The strict INT96/utc-timezone gates were a deliberate decision: physical-encoding differences a reader can observe fall back rather than silently diverge.