Skip to content

[flink] Support Blob tables in dedicated split generation - #8914

Open
wwj6591812 wants to merge 1 commit into
apache:masterfrom
wwj6591812:agent/fix-dedicated-blob-split-generation
Open

[flink] Support Blob tables in dedicated split generation#8914
wwj6591812 wants to merge 1 commit into
apache:masterfrom
wwj6591812:agent/fix-dedicated-blob-split-generation

Conversation

@wwj6591812

Copy link
Copy Markdown
Contributor

Purpose

Large batch scans can enable scan.dedicated-split-generation to move split planning and emission from the JobManager source coordinator to a dedicated TaskManager operator. This avoids retaining and checkpointing a very large set of split assignments in the JobManager for tables with millions of files.

However, the dedicated reader path did not preserve Blob conversion semantics. ReadOperator always wrapped rows with FlinkRowData, so an internal Blob value was read as byte[] and failed with ClassCastException. This prevented Blob tables from using dedicated split generation.

Changes

  • Propagate the physical read type and blob-as-descriptor option from FlinkSourceBuilder through MonitorSource to ReadOperator.
  • Use FlinkRowDataWithBlob when the projected read type contains BLOB, ARRAY<BLOB>, or MAP<*, BLOB> fields.
  • Keep the existing constructors and MonitorSource.buildSource overload for compatibility.
  • Preserve the existing outer projection and field reordering behavior.

Tests

  • Added ReadOperatorBlobTest for raw Blob bytes and serialized descriptors.
  • Added a Flink integration test covering scan.dedicated-split-generation=true, raw/descriptor modes, projection, and field reordering.
  • Flink 1 profile: 3 targeted tests passed, plus 7 related regression tests.
  • Flink 2 profile: the same 3 targeted tests passed.
  • Checkstyle, Spotless, and git diff --check passed.

@wwj6591812
wwj6591812 marked this pull request as ready for review July 29, 2026 17:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant