Skip to content

Iceberg incremental tests#3

Merged
kraftaa merged 5 commits intoicebergfrom
iceberg-incremental-tests
Mar 8, 2026
Merged

Iceberg incremental tests#3
kraftaa merged 5 commits intoicebergfrom
iceberg-incremental-tests

Conversation

@kraftaa
Copy link
Copy Markdown
Owner

@kraftaa kraftaa commented Feb 26, 2026

Summary

This PR adds incremental-correctness test coverage on top of the Iceberg branch, focused on failure/restart safety and duplicate-watermark paging behavior.

What’s Included

  1. Duplicate watermark paging test
  • Verifies composite cursor pagination (watermark + tiebreaker) does not skip rows when many records share the same watermark value.
  1. State migration test
  • Verifies legacy watermarks schema (without cursor_value) is migrated on open and existing progress is preserved.
  1. Resume-from-progress test
  • Verifies sync resumes from saved progress and writes only remaining rows.
  1. Missing cursor state guard test
  • Verifies cursor mode fails fast when state has watermark but missing cursor.
  1. Unique watermark mode test
  • Verifies incremental_column_is_unique: true works without tiebreaker and persists progress.
  1. Test intent comments
  • Added one-line comments above new tests for quick reviewer context.

Files Changed

  • src/reader.rs
  • src/state.rs
  • src/sync.rs

New/Updated Tests

  • read_batch_cursor_paging_handles_duplicate_watermarks
  • open_migrates_legacy_watermarks_table
  • sync_table_resumes_from_saved_progress
  • sync_table_fails_when_cursor_state_missing_in_cursor_mode
  • sync_table_supports_unique_watermark_without_tiebreaker

Validation

  • cargo fmt --all
  • cargo clippy --all-targets -- -D warnings
  • cargo test (all passing)

Notes

  • DB-backed integration-style tests are env-gated via RUSTREAM_IT_DB_URL and no-op when not set.
  • No production behavior changes intended in this PR; scope is test coverage and test documentation.

@kraftaa kraftaa changed the base branch from main to iceberg March 8, 2026 01:39
@kraftaa kraftaa merged commit 00b8c07 into iceberg Mar 8, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant