What needs to happen?
Beam has an incremental append source for Iceberg, but it ignores overwrite snapshots.
Add a full CDC source that tracks all relevant snapshots and outputs Beam Rows with CDC Value Kinds. It should support all existing features (row filtering, column pruning, reading between snapshot boundaries), along with new features like outputting metadata columns and watermarking by a specific timestamp column
Design doc: https://s.apache.org/beam-iceberg-incremental-cdc-source
Prototype: #38619
Issue Priority
Priority: 2 (default / most normal work should be filed as P2)
Issue Components
What needs to happen?
Beam has an incremental append source for Iceberg, but it ignores overwrite snapshots.
Add a full CDC source that tracks all relevant snapshots and outputs Beam Rows with CDC Value Kinds. It should support all existing features (row filtering, column pruning, reading between snapshot boundaries), along with new features like outputting metadata columns and watermarking by a specific timestamp column
Design doc: https://s.apache.org/beam-iceberg-incremental-cdc-source
Prototype: #38619
Issue Priority
Priority: 2 (default / most normal work should be filed as P2)
Issue Components