Skip to content

feat: Persist destination write stream state#874

Open
1neoneo3 wants to merge 1 commit into
supabase:mainfrom
1neoneo3:feature/bigquery-append-only-history
Open

feat: Persist destination write stream state#874
1neoneo3 wants to merge 1 commit into
supabase:mainfrom
1neoneo3:feature/bigquery-append-only-history

Conversation

@1neoneo3

@1neoneo3 1neoneo3 commented Jul 3, 2026

Copy link
Copy Markdown

What

Adds durable destination write stream state to the replication store.

This PR introduces a small store-level foundation that lets a destination persist the state of an external write stream, including:

  • the destination stream name
  • the next offset to use
  • creation and update timestamps

The implementation is wired through:

  • the Postgres-backed store
  • the in-memory store
  • the notifying test store wrapper
  • the error-reporting state store wrapper
  • a Postgres migration for the new destination_write_streams table
  • store-level tests for create/read/update/reset behavior

Why

BigQuery Storage Write API committed streams need a stable stream name and durable offsets to make retries safe.

Without persisted offsets, a destination can append the same logical rows again after a retry or process restart. This state store is the foundation needed by #875 to use BigQuery committed streams safely for append-only history writes.

Relationship to the follow-up PR

This is the first PR in the split for #873.

Keeping this as a separate PR makes the storage contract reviewable without the larger BigQuery append-only implementation in the same diff.

Behavior and compatibility

  • This PR does not change existing destination write behavior by itself.
  • Existing destinations are not required to use the new state API.
  • The new Postgres table is additive.
  • In-memory and notifying stores are updated so existing tests and future destination tests can use the same store contract.
  • Reset-for-resync clears in-memory destination write stream state, matching the Postgres store behavior.

Tests

  • cargo check -p etl --features test-utils
  • cargo check -p etl-replicator --no-default-features
  • cargo test -p etl --features test-utils reset_for_resync_clears_destination_write_stream_state --test main

Refs #873

@1neoneo3 1neoneo3 changed the title Add BigQuery append-only history mode feat: Add BigQuery append-only history mode Jul 3, 2026
@1neoneo3 1neoneo3 force-pushed the feature/bigquery-append-only-history branch from fc12b4a to be5d594 Compare July 3, 2026 14:13
@1neoneo3 1neoneo3 changed the title feat: Add BigQuery append-only history mode feat: BigQuery append-only history modeを追加 Jul 3, 2026
@1neoneo3 1neoneo3 changed the title feat: BigQuery append-only history modeを追加 feat: Add BigQuery append-only history mode Jul 3, 2026
@1neoneo3 1neoneo3 force-pushed the feature/bigquery-append-only-history branch from be5d594 to 494b203 Compare July 3, 2026 14:23
@1neoneo3 1neoneo3 changed the title feat: Add BigQuery append-only history mode feat: Persist destination write stream state Jul 3, 2026
@1neoneo3 1neoneo3 marked this pull request as ready for review July 3, 2026 14:24
@1neoneo3 1neoneo3 requested a review from a team as a code owner July 3, 2026 14:24

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 494b203d33

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/etl/src/store/state/base.rs
Comment thread crates/etl/src/store/both/postgres.rs
@1neoneo3 1neoneo3 force-pushed the feature/bigquery-append-only-history branch 2 times, most recently from cec3718 to 9bde359 Compare July 3, 2026 14:43
@1neoneo3

1neoneo3 commented Jul 3, 2026

Copy link
Copy Markdown
Author

This PR is the first foundation slice for #873. It only adds durable destination write stream state and does not enable the BigQuery append-only mode by itself. #875 is stacked on top as the follow-up implementation.

@1neoneo3 1neoneo3 force-pushed the feature/bigquery-append-only-history branch from 9bde359 to e273987 Compare July 4, 2026 00:58
@iambriccardo

Copy link
Copy Markdown
Contributor

Thanks for the PR! We are currently thinking internally on how we want to add arbitrary metadata for destinations to store. Since it's hard to predict what each destination will need, we were thinking of going towards a design where any destination can store arbitrary data into the store.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants