Skip to content

feat(THU-596): user data export & import#991

Open
raivieiraadriano92 wants to merge 13 commits into
mainfrom
raivieiraadriano92/thu-596-export-user-data-export-ships-before-workspaces-v1
Open

feat(THU-596): user data export & import#991
raivieiraadriano92 wants to merge 13 commits into
mainfrom
raivieiraadriano92/thu-596-export-user-data-export-ships-before-workspaces-v1

Conversation

@raivieiraadriano92

@raivieiraadriano92 raivieiraadriano92 commented Jun 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Adds Export My Data and Import Data actions to Settings → Preferences → Data: one-click versioned JSON snapshot of every user-content table, plus restore via SELECT + UPDATE/INSERT inside a single transaction (works on PowerSync's view-backed synced tables, where INSERT ... ON CONFLICT DO UPDATE is rejected).
  • File format is versioned (schemaVersion: 1); v1 ships without a workspace concept so it can land before Workspaces v1. The post-workspaces refactor (THU-597) bumps to schemaVersion: 2 with workspaceId on rows and no selector UI — spec in docs/architecture/export-format.md.
  • Schema-driven: both halves walk includedTables derived from src/db/powersync/schema.ts; adding a new synced/local-only table flows through automatically with the export's allowlist test as the safety net.

Test plan

  • bun test src/dal/export.test.ts src/dal/import.test.ts src/lib/export-download.test.ts src/lib/import-upload.test.ts all green (~40 tests)
  • bun tsc --noEmit, bun lint, bun run format-check all clean
  • Sign in, seed chats / tasks / a custom model with an API key → Export My Data downloads thunderbolt-export-YYYY-MM-DD.json with schemaVersion: 1 and the expected table buckets
  • Open the file: tables.integrations_secrets / tables.devices / tables.agents_system absent; models_secrets contains the typed API key
  • Sign out → sign back in (fresh DB) → Import Data → pick the file → confirmation dialog shows row count + source email + export date → confirm → success message, chats / tasks / models all restored
  • Re-import on top of the populated DB → imported file wins on overlapping IDs, no duplicates; non-overlapping local rows untouched
  • Malformed file ({} or wrong schemaVersion) → inline error, no DB writes; tampered file containing devices / integrations_secrets lands in ignoredTableNames

Note

High Risk
Exports plaintext API keys and MCP/agent secrets; import overwrites matching IDs locally and propagates to all synced devices, with destructive restore semantics.

Overview
Adds Export My Data and Import Data under Settings → Preferences → Data for signed-in users, backed by a versioned thunderbolt-export JSON envelope (schemaVersion: 1).

Export walks PowerSync’s syncedTables + localOnlyTables (now exported from schema.ts), minus devices, integrations_secrets, and agents_system, and snapshots full rows—including soft-deleted rows and local credential tables (models_secrets, mcp_secrets, agents_secrets). Downloads use downloadJson with a dated filename.

Import validates the envelope, previews via summarizeExportEnvelope (row count, source email, cross-account mismatch), then restores in one transaction using per-row SELECT + UPDATE/INSERT (avoids upserts on PowerSync views). Imported rows win on PK collision; other local rows stay put. Synced tables get userId re-stamped from the session; unknown table keys are ignored. File pick goes through readJsonFile with a 200 MB cap.

docs/architecture/export-format.md documents the format and import semantics (including multi-device sync overwrite). PostHog events settings_data_export / settings_data_import were added.

Reviewed by Cursor Bugbot for commit 2f5b997. Bugbot is set up for automated code reviews on this repo. Configure here.

@raivieiraadriano92 raivieiraadriano92 self-assigned this Jun 16, 2026
@github-actions

Copy link
Copy Markdown

Semgrep Security Scan

No security issues found.

@github-actions

github-actions Bot commented Jun 16, 2026

Copy link
Copy Markdown

Preview environment deployed 🚀

Service URL
Marketing / blog / docs https://thunderbolt-pr-991.preview.thunderbolt.io
App https://app-pr-991.preview.thunderbolt.io
API https://api-pr-991.preview.thunderbolt.io
Keycloak https://auth-pr-991.preview.thunderbolt.io
PowerSync https://powersync-pr-991.preview.thunderbolt.io

Stack: preview-pr-991 · Commit: 2f5b997ab71c0cfeb9f52c0623f8b02911129f35

Auto-destroys on PR close/merge. Login via the bundled Keycloak realm — demo@thunderbolt.io / demo by default.

@github-actions

github-actions Bot commented Jun 16, 2026

Copy link
Copy Markdown

PR Metrics

Metric Value
Lines changed (prod code) +839 / -5
JS bundle size (gzipped) 🟢 682.3 KB → 683.3 KB (+1.0 KB, +0.1%)
Test coverage 🟢 78.09% → 78.29% (+0.2%)
Performance (preview) Preview not ready — Render deploy may have timed out
Accessibility
Best Practices
SEO

Updated Sun, 21 Jun 2026 14:21:29 GMT · run #1978

@raivieiraadriano92 raivieiraadriano92 changed the title feat(THU-596): user data export feat(THU-596): user data export & import Jun 17, 2026
@raivieiraadriano92 raivieiraadriano92 marked this pull request as ready for review June 17, 2026 13:09

@ital0 ital0 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 thunder-deep-review — advisory (pending; submit or discard after review).

CONCERNS — core feature works, but import writes untrusted/verbatim rows into synced tables, bypassing DAL invariants and silently propagating cross-device (and potentially cross-account/E2EE) with no size/column guards; several blocking architecture+powersync+security items plus a wrong forward-compat doc claim should be resolved before merge.

5 blocking · 7 convention · 17 nit

All 5 blocking + 7 convention findings are attached as inline comments. The 17 nits:

  • src/dal/import.ts:147-152 — N+1 write pattern: per-row SELECT then UPDATE/INSERT, serial inside one tx; batch-load existing PKs with a single IN(...) SELECT into a Set
  • src/dal/import.ts:149 — UPDATE writes set(row) including the PK column; omit pk from the SET payload
  • src/dal/export.ts:91-93 — Export materializes every row of every table via Promise.all then pretty JSON.stringify (~2x); large transient memory spike, no streaming/pagination
  • src/settings/preferences.tsx:376 — handleConfirmImport clears importError but never importSuccess; stale success banner + new error can render together
  • src/settings/preferences.tsx:877 — Export/Import UI gated on isAuthenticated (anonymous too) though component derives isFullUser; copy implies a connected account; gate vs copy vs cross-user provenance
  • src/dal/import.ts:132 — Recognized table whose value is a non-array is silently skipped by the empty-array branch (not counted, not ignored); rows-not-object inside a valid array throw — inconsistent contract
  • src/settings/preferences.tsx:939 — aria-describedby resolves to the success element id after import, so the button is permanently described by the last result instead of its instructional description
  • src/settings/preferences.tsx:362 — Inline 7-line sourceEmail type-guard ternary breaks symmetry with its extracted siblings; double cast appears twice — extract a readSourceEmail helper
  • src/dal/import.ts:39 — Single-use candidates const split from the .find on the next line; collapse into one chained expression
  • src/db/tables.ts:315 — Comment on agentsSecretsTable says 'Never leaves the device' but this PR's export now includes agents_secrets — invariant no longer true (NOT in diff → body)
  • docs/architecture/export-format.md:82 — Grammar: 'a ImportFormatError' → 'an ImportFormatError'
  • docs/architecture/export-format.md:87 — Cites multi-device-sync.md for 'no FK constraints' but that doc only says 'No composite foreign keys'; weaken wording or drop citation
  • src/settings/preferences.tsx:1448 — Dialog 'contains {totalRows} rows' counts every bucket (incl. excluded/unknown) while success toast 'Imported {total} rows' sums only upserted — numbers silently disagree
  • src/dal/export.ts:13 — Widening syncedTables/localOnlyTables const→export to feed export.ts removes a guardrail; add a comment marking them the canonical classification source
  • docs/architecture/export-format.md:44-46 — New SYNCED table is auto-included in export AND import-with-upload with zero sync-classification review; note + import test to mirror the export allowlist gate
  • src/dal/import.test.ts:284-292 — derivePkSpec tests suppress console.warn with a hand-rolled save/restore; use spyOn(console,'warn').mockImplementation(()=>{}) per R-SUPPRESSCONSOLE
  • src/settings/preferences.tsx:67-68 — New reducer actions modeled as setters (SET_IS_EXPORTING) not events; R-REDUCER prefers EXPORT_STARTED/FINISHED (low priority, file is locally consistent)

Note: the src/db/tables.ts:315 nit is outside the PR diff, so it could not be anchored inline — listed above only.

Comment thread src/dal/import.ts Outdated
Comment thread src/dal/import.ts Outdated
Comment thread src/dal/import.ts
Comment thread src/settings/preferences.tsx Outdated
Comment thread docs/architecture/export-format.md Outdated
Comment thread src/settings/preferences.tsx Outdated
Comment thread src/lib/import-upload.ts
Comment thread src/settings/preferences.tsx Outdated
Comment thread src/dal/export.ts
Comment thread src/dal/import.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants