[core][flink][spark] Refresh data evolution vector indexes incrementally

[leaves12138]

Purpose

Data-evolution append tables can partially update an indexed vector column, for example by filling an embedding from NULL to a value. Incremental global-index builds previously considered only uncovered row ranges, so an existing vector index could remain readable but permanently miss those updated vectors unless users performed a full rebuild.

This change refreshes only affected logical vector-index ranges while keeping the old index active until its replacement is ready.

Changes

• Store the data snapshot scanned by every generic global-index build as scanSnapshotId in GlobalIndexMeta.sourceMeta, encoded with a DEIX magic number and version. Source metadata is recorded unconditionally once the build has a scan snapshot, independent of scalar/vector index type.
• Keep source tracking separate from automatic refresh: scalar indexes record the watermark but retain their existing build behavior; only supported vectors on data-evolution tables enter the refresh planner.
• Deliberately record the scan snapshot rather than the later index-commit snapshot, so data updates committed while an index build is running remain newer than the recorded watermark and are picked up by the next build.
• Select an existing vector-index range for refresh when an active data file intersects its row range, physically contains an indexed field resolved by stable field ID, and has maxSequenceNumber newer than scanSnapshotId.
• Refresh legacy vector indexes without this metadata once.
• Merge refreshed ranges with newly uncovered ranges in the Flink and Spark builders.
• Commit old-index deletion and replacement-index addition atomically.
• Detect concurrent replacements by requiring the expected old index file to remain active and by preventing an older scan generation from replacing a newer one.
• Require a non-null scan snapshot whenever a generic Flink index build produces work.

User impact

Users can continuously populate embedding columns and run incremental index builds. Search continues using the previous complete index during construction, then switches atomically to the more complete replacement without an index-unavailable window. Scalar generic indexes also carry the scan watermark for future type-specific handling, without changing their refresh behavior.

Tests

• Core data-evolution metadata, planner, manifest, and conflict-detection suites passed.
• Flink generic topology and scalar full-text suites: 34 passed.
• Spark scalar full-text suite: 7 passed.
• Spark Lumina native suite on Linux: 10 passed.
• Downstream Morax topology and vector suites on Linux: 35 passed, 1 existing platform-assumption skip.
• Spotless and git diff --check passed.

[JingsongLi]

Older versions of the vector index are skipped by the refresh planner when DEIX source metadata is missing; at the same time, the row ranges covered by the index are not added to unindexedRowRanges, resulting in the first automatic refresh—as described in the PR—never being triggered after the upgrade.

[JingsongLi]

The feature “automatically refresh the old vector index once after an upgrade” has not yet been implemented**. The old index lacks sourceMeta, so the current planner skips it entirely; however, the range planner treats the old index as overwritten, so it is neither rebuilt nor included in the incremental refresh. New tests even explicitly assert that the old index “does not automatically refresh,” which contradicts the PR description and previous feedback. It is recommended that this change not be merged for now.