Skip to content

Sync FxA DB mirror schemas with upstream migrations (patches 145-192)#9339

Draft
clouserw wants to merge 1 commit into
mozilla:mainfrom
clouserw:fxa-update-schema
Draft

Sync FxA DB mirror schemas with upstream migrations (patches 145-192)#9339
clouserw wants to merge 1 commit into
mozilla:mainfrom
clouserw:fxa-update-schema

Conversation

@clouserw
Copy link
Copy Markdown
Member

@clouserw clouserw commented May 8, 2026

Description

Update BigQuery external table definitions for the FxA database mirrors (both prod and stage) to reflect schema changes from FxA DB migrations 145 through 192 (Nov 2023 - May 2026). I don't know of a way to test this.. Thanks

Reviewer, please follow this checklist

@clouserw
Copy link
Copy Markdown
Member Author

clouserw commented May 8, 2026

This is essentially the next iteration of #4605 . @akkomar @sean-rose Thank you

Copy link
Copy Markdown
Member

@akkomar akkomar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't dry-run external queries, but I tried testing them with limit 1 - some of them fail because of missing permissions.
We'd need to grant these in prod and stage:

  -- new tables (column-list grants)
  GRANT SELECT (domain, createdAt) ON `domainBlocklist` TO 'bigquery_ro';
  GRANT SELECT (regex,  createdAt) ON `emailBlocklist`  TO 'bigquery_ro';
  GRANT SELECT (
    uid, credentialId, signCount, transports, aaguid, name,
    createdAt, lastUsedAt, backupEligible, backupState, prfEnabled
  ) ON `passkeys` TO 'bigquery_ro';

  -- new columns on existing column-granted tables
  GRANT SELECT (verificationMethod) ON `accountResetTokens`   TO 'bigquery_ro';
  GRANT SELECT (verificationMethod) ON `passwordForgotTokens` TO 'bigquery_ro';

@clouserw this might require following process from https://mozilla-hub.atlassian.net/browse/SVCSE-2521?focusedCommentId=1015649.

@clouserw
Copy link
Copy Markdown
Member Author

Thanks for testing. I've filed SREIN-1291 to grant the extra permissions.

@clouserw clouserw marked this pull request as draft May 12, 2026 01:16
Update BigQuery external table definitions for the FxA database mirrors
(both prod and nonprod/stage) to reflect schema changes from FxA DB
migrations 145 through 192 (Nov 2023 - May 2026).

Modified tables:
- carts: add currency, stripeSubscriptionId, stripeIntentId, isFreeTrial
- securityEvents: add additionalInfo (JSON)
- accountResetTokens: add verificationMethod
- passwordForgotTokens: add verificationMethod

New tables:
- deletedAccounts: tracks deleted account UIDs and timestamps
- passkeys: WebAuthn/FIDO2 credential metadata
- emailBlocklist: regex-based email blocking rules
- domainBlocklist: domain-based blocking rules

No changes needed for fxa_oauth or fxa_profile databases (no migrations
since before Nov 2023). Credential columns (clientSalt, verifyHashVersion2,
wrapWrapKbVersion2, publicKey) intentionally excluded from mirror.
@clouserw clouserw force-pushed the fxa-update-schema branch from 5fe9bc2 to 69efc15 Compare May 12, 2026 14:18
@clouserw
Copy link
Copy Markdown
Member Author

We can't add permissions until after the table exists, so, I've removed the passkeys tables from this patch. The others will exist after our push on Wednesday (tomorrow). So, I'm going to wait until then, then reopen the permissions issue, once that lands we can come back to this. So, marking as a draft until tomorrow. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants