Skip to content

feat(provider): add MigrationPolicy / ProviderConfig for VerifyOnly construction#10

Merged
affandar merged 7 commits into
microsoft:mainfrom
pinodeca:feat/migration-policy
May 24, 2026
Merged

feat(provider): add MigrationPolicy / ProviderConfig for VerifyOnly construction#10
affandar merged 7 commits into
microsoft:mainfrom
pinodeca:feat/migration-policy

Conversation

@pinodeca

@pinodeca pinodeca commented May 20, 2026

Copy link
Copy Markdown
Contributor

Introduces a ProviderConfig / MigrationPolicy API for PostgresProvider construction, with related safety hardening for the migration path. Backend processes that must not run DDL can construct with MigrationPolicy::VerifyOnly; the existing default (MigrationPolicy::ApplyAll) preserves today's behavior.

VerifyOnly does not take the migration advisory lock and does not create or modify any database objects, eliminating per-connection lock contention and DDL-grant requirements for backend processes.

Commits

715ec60 feat(provider): add MigrationPolicy / ProviderConfig for VerifyOnly construction

The original PR commit. Introduces MigrationPolicy { ApplyAll, VerifyOnly }, ProviderConfig, and the new constructor surface. ApplyAll is the default and preserves pre-feature behavior across new, new_with_schema, and the Entra constructors. VerifyOnly verifies that _duroxide_migrations exists in the target schema and that every bundled migration has been applied, returning an error otherwise.

17898d0 feat(migrations): reject unknown applied migrations

Closes a foot-gun where an older binary could silently rewrite a database that is ahead of its code. MigrationRunner now refuses to proceed when the tracking table records versions the running binary does not bundle:

  • Under ApplyAll the check runs under the migration advisory lock and before any DDL, so a downgrade can never half-rewrite an ahead-of-code schema. The error names the unknown versions and instructs the operator to update the code.
  • Under VerifyOnly the same check runs as part of verification.

92d476c refactor(provider)!: collapse constructors into new_with_config

API cleanup before stabilizing the new surface. Collapses the various *_with_config and Entra-specific constructors into a single PostgresProvider::new_with_config(ProviderConfig). ProviderConfig now carries the connection variant via a new ConnectionConfig enum (Url(String) or Entra { host, port, database, user, options }), the optional schema name, and the migration policy.

Construct via ProviderConfig::url(database_url) or ProviderConfig::entra(host, port, db, user, options) and adjust fields as needed. The previous URL/Entra constructors are kept as #[deprecated] shims that delegate to new_with_config, so callers on 0.1.x source paths continue to compile. Breaking only for the unreleased new_with_config(url, cfg) shape introduced in 715ec60.

859b062 feat(provider)!: validate schema names

Adds a conservative validator (^[A-Za-z_][A-Za-z0-9_]*$) called from new_with_config — the single entry point all other constructors now delegate to.

PostgreSQL identifiers cannot be bound as SQL parameters, so the schema name is interpolated directly into the DDL and DML the provider issues (CREATE SCHEMA {schema}, INSERT INTO {schema}._duroxide_migrations ..., etc.). Without validation, a caller that forwards attacker-controlled input as schema_name would have a SQL injection vector. PostgreSQL's full identifier grammar is broader; this validation is intentionally conservative. Already-shipped releases (<= 0.1.33) are unaffected; the break is on unreleased API surface only.

d5d1619 test(provider): port remaining opt initialization tests

Pure additive. Adds four integration tests against the new API covering provider-initialization paths that were previously implicit:

  • verify_only_errors_when_schema_missingVerifyOnly against a database where the target schema does not exist at all.
  • verify_only_errors_when_tracking_table_missingVerifyOnly against a bare schema (CREATE SCHEMA was run, but no migrations applied).
  • verify_only_errors_when_migrations_behindVerifyOnly against a schema whose tracking table is missing the most recent bundled migration; the error names the missing version.
  • concurrent_apply_all_is_serialized — two concurrent ApplyAll initializations against the same fresh schema must both succeed, exercising the migration advisory lock end-to-end.

Squash strategy

Commits stay split for review. At merge time they squash cleanly along two seams: 17898d0 is a self-contained safety fix, and 92d476c + 859b062 form one breaking-API commit. d5d1619 is pure tests and can squash with either of the above. Happy to reorganize before merge if preferred.

Tests

  • cargo fmt --check
  • cargo clippy --tests ✅ (no new warnings on touched code; pre-existing warnings in session_e2e_tests and src/provider.rs:3121 are unrelated)
  • cargo test --test migration_policy_tests ✅ (10 passed)
  • Library tests (cargo test --lib) ✅
  • Regression / concurrent-migration / cached-plan-retryable tests ✅

…onstruction

Introduces an optional ProviderConfig accepted by two new constructors:
- PostgresProvider::new_with_config(url, config)
- PostgresProvider::new_with_schema_and_config(url, schema, config)

ProviderConfig::migration_policy controls startup behavior:
- ApplyAll (default): apply pending embedded migrations, identical to today.
- VerifyOnly: do not run DDL; verify _duroxide_migrations exists and every
  bundled migration has been applied. Intended for processes that must not
  modify schema -- e.g. application backends, when a separately privileged
  worker is responsible for migrations.

VerifyOnly does not take the migration advisory lock and does not create or
modify any database objects, eliminating per-connection lock contention and
DDL-grant requirements for backend processes.

Existing constructors (new, new_with_schema, new_with_entra,
new_with_schema_and_entra) are unchanged at the source and behavior level;
new_with_schema is refactored to delegate to new_with_schema_and_config with
ProviderConfig::default(), preserving ApplyAll semantics.

New types are re-exported from the crate root. Added integration tests in
tests/migration_policy_tests.rs covering ApplyAll, successful VerifyOnly
against an initialized schema, and VerifyOnly's error path against an
uninitialized schema.
pinodeca and others added 5 commits May 20, 2026 15:46
Both MigrationPolicy::ApplyAll and MigrationPolicy::VerifyOnly now fail
fast when the _duroxide_migrations tracking table records versions that
are not bundled with the running binary (schema is ahead of the code).

Under ApplyAll the check runs under the migration advisory lock and
short-circuits before any DDL is executed, so an older binary cannot
rewrite a schema that is ahead of its code. Under VerifyOnly the check
runs between the tracking-table-exists check and the missing-versions
check, producing a distinct ahead-of-code error instead of a misleading
verification success.

Tests added (ported from duroxide-pg-opt):
- verify_only_rejects_unknown_migrations: VerifyOnly errors when an
  unknown version is present in the tracking table.
- apply_all_unknown_migrations_causes_no_mutations: ApplyAll errors when
  an unknown version is present AND no DDL is executed (asserts the
  previously-deleted real migration is still absent and the
  previously-dropped core table is still missing).
Consolidate the provider construction surface to a single
config-driven entry point, ported from duroxide-pg-opt:

  PostgresProvider::new_with_config(ProviderConfig)

ProviderConfig now carries the connection variant via a new
ConnectionConfig enum (Url(String) or Entra { host, port, database,
user, options }), plus the optional schema name and the migration
policy. Construct via ProviderConfig::url(database_url) or
ProviderConfig::entra(host, port, db, user, options) and adjust
fields as needed.

Breaking changes (unreleased API surface only):
- Removed new_with_config(url, ProviderConfig) and
  new_with_schema_and_config(url, schema, ProviderConfig). These
  shipped only on this feature branch and were never released.

Compatibility:
- new(url) and new_with_schema(url, schema) remain as thin
  convenience wrappers around the new path.
- new_with_entra and new_with_schema_and_entra are deprecated and
  delegate to new_with_config(ProviderConfig::entra(...)). They will
  be removed in a future release.
- initialize_schema is deprecated; every constructor already runs
  the migration runner.

Internal changes:
- new_from_url is a private URL-path helper.
- new_with_entra_with_token_source (the crate-internal test seam)
  now also takes MigrationPolicy, so VerifyOnly applies to the
  Entra path too. The three internal pipeline tests pass ApplyAll.

Tests:
- tests/migration_policy_tests.rs migrated to the new API.
- tests/entra_live_test.rs migrated to the new API.

Verified: cargo build --tests clean, cargo fmt --check clean,
cargo clippy --tests produces no new warnings, all 38 lib tests,
5 migration_policy_tests, 2 regression_tests, 2 concurrent_migration
tests, 1 cached_plan_retryable test, plus basic_tests all pass.
Add a conservative schema-name validator and call it from
PostgresProvider::new_with_config (the single entry point all other
constructors now delegate to). Schema names that do not match
[A-Za-z_][A-Za-z0-9_]* are rejected up front with a clear error.

Why a validator is needed:

PostgreSQL identifiers cannot be bound as SQL parameters, so the
schema name is interpolated directly into the DDL and DML the
provider issues (e.g. CREATE SCHEMA {schema}, INSERT INTO
{schema}._duroxide_migrations ...). Without validation, a caller that
forwards attacker-controlled input as schema_name would create a SQL
injection vector. PostgreSQL's full identifier grammar (including
quoted identifiers) is broader; this validation is intentionally
conservative.

Breaking changes (unreleased API surface only):
- new_with_config, new_with_schema, and the deprecated Entra
  constructors now return Err for schema names outside the accepted
  subset. Already-shipped releases (<= 0.1.33) are unaffected.
- Callers passing only constants from their own code are unaffected.

Tests:
- schema_name_validation_rejects_unsafe_identifiers exercises the
  rejection path via new_with_config("bad-name").

Verified: cargo build --tests clean, cargo fmt --check clean,
cargo clippy --tests clean on touched files, all 6
migration_policy_tests pass.
Add four integration tests against the new ProviderConfig /
new_with_config API, covering provider initialization paths that
were previously implicit:

- verify_only_errors_when_schema_missing: VerifyOnly against a
  database where the target schema does not exist at all.
- verify_only_errors_when_tracking_table_missing: VerifyOnly against
  a bare schema (CREATE SCHEMA was run, but no migrations applied)
  must reject with a clear "not initialized" error.
- verify_only_errors_when_migrations_behind: VerifyOnly against a
  schema whose tracking table is missing the most recent bundled
  migration must name the missing version.
- concurrent_apply_all_is_serialized: two concurrent ApplyAll
  initializations against the same fresh schema must both succeed,
  exercising the migration advisory lock end-to-end.

These are the equivalents of the tests added in the optimistic
sibling crate, adapted to this provider's actual API
(ProviderConfig::url + new_with_config) and error messages.

Pure additive: no library code changes.

Verified: cargo fmt --check clean, all 10 migration_policy_tests
pass (6 pre-existing + 4 new). Pre-existing clippy warnings in
other test targets and src/provider.rs are unrelated.
The 0.1.x series is not enforcing strict SemVer; this release will ship
as 0.1.34 instead of 0.2.0. Update deprecation attributes on
new_with_entra, new_with_schema_and_entra, and initialize_schema to
match. Also trims a now-redundant comment on initialize_schema.
@pinodeca

Copy link
Copy Markdown
Contributor Author

Findings by GPT-5.5:

  • P1: VerifyOnly can accept a schema with no runtime tables.
    In duroxide-pg/src/migrations.rs, verify() only checks that _duroxide_migrations exists and contains every embedded migration version. It never checks that the actual provider tables still exist. That matters because migrate_inner() already has a repair path for exactly this corrupted state, “migration marked applied but tables don’t exist,” via check_tables_exist() in duroxide-pg/src/migrations.rs. With VerifyOnly, the same schema would construct successfully and then fail later on normal provider operations with missing relation/function errors. Add a VerifyOnly check for at least the key tables, and ideally the required stored procedures too. A focused regression would be: apply migrations, drop instances, then assert VerifyOnly fails.

  • P2: The new schema-name validator accepts identifiers that the unquoted SQL path does not handle consistently.
    duroxide-pg/src/provider.rs allows uppercase schema names via ^[A-Za-z_][A-Za-z0-9_]*$, but all schema references are emitted unquoted. PostgreSQL folds unquoted MySchema to myschema, while the migration verifier checks information_schema.tables.table_schema = $1 using the original mixed-case string in duroxide-pg/src/migrations.rs and duroxide-pg/src/migrations.rs. Result: ApplyAll can create/use myschema, but VerifyOnly against MySchema reports the migration table missing, and a later ApplyAll can think tables are absent and try to reapply migrations. Either canonicalize accepted schema names to lowercase before storing them, or reject uppercase identifiers. Reserved words are a related edge case for the same reason: they pass validation but can fail in unquoted SQL.

Notes

I did not run tests, per your request. The new test suite covers the main migration-policy paths well, but it currently misses both “tracking table present, runtime tables absent” and mixed-case schema-name behavior.

@affandar affandar merged commit 94eed2c into microsoft:main May 24, 2026
4 checks passed
pinodeca added a commit to microsoft/pg_durable that referenced this pull request May 26, 2026
Drop the duroxide-pg-opt git submodule and depend on the upstream duroxide-pg crate directly. Adopt the new ProviderConfig / MigrationPolicy API: backends use VerifyOnly (no DDL, no advisory locks), the background worker uses ApplyAll.

Temporarily pins the dependency to a fork branch backing microsoft/duroxide-pg#10 until that PR merges and a tag is published.

- Remove .gitmodules and duroxide-pg-opt submodule
- Cargo.toml: cargo git dep (fork branch, with TEMP comment)
- Update all PostgresProvider call sites to new_with_schema_and_config(url, Some(DUROXIDE_SCHEMA), config)
- types.rs: backend_provider_config / worker_provider_config helpers
- Drop GH_PAT plumbing in CI/devcontainer (no longer needed)
- Update docs and prompts to reference duroxide-pg
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants