Skip to content

Add SupaMail v0.1 — IMAP mirror to Postgres/Supabase#5

Open
fedster99 wants to merge 11 commits into
mainfrom
fedster99/supamail-v0.1
Open

Add SupaMail v0.1 — IMAP mirror to Postgres/Supabase#5
fedster99 wants to merge 11 commits into
mainfrom
fedster99/supamail-v0.1

Conversation

@fedster99
Copy link
Copy Markdown
Owner

Summary

  • First release of SupaMail: a worker (+ optional API) that mirrors IMAP folders, messages, MIME bodies, and attachment metadata into Supabase/Postgres with sync-health bookkeeping. Default deploy path is Fly.io worker + Supabase Postgres.
  • Schema lives in supabase/migrations/0001_imap_mirror.sql — seven tables under public.imap_*, RLS enabled with no policies (deny-all to anon/authenticated by design), session-affine DATABASE_URL required because reconciliation relies on advisory locks.
  • Built-in smoke tests: pnpm dry-run:local runs against a fake IMAP fixture, pnpm smoke:greenmail spins a real greenmail/standalone Docker IMAP server and syncs through the protocol.

Things to look at first (non-obvious decisions)

  • Encryption is Node-side, not pgcrypto. encrypted_password is AES-256-GCM ({ version | iv | tag | ciphertext }) computed in src/crypto.ts. The migration DROPs the legacy imap_encrypt_password / imap_decrypt_password SQL functions so the master key never crosses into Postgres logs or pg_stat_statements.
  • SSRF guard on IMAP target. src/host-validation.ts rejects private/link-local/loopback/metadata IPs (including IPv4-mapped IPv6), restricts ports to 143/993, and refuses plaintext IMAP unless IMAP_ALLOW_PRIVATE_HOSTS=true. Local dev and the smoke scripts opt in via per-script config override.
  • Bearer auth is crypto.timingSafeEqual on equal-length buffers, not !==. POST /migrate falls back to API_TOKEN if ADMIN_TOKEN is unset.
  • Reconciliation streams UIDs into a TEMP TABLE … ON COMMIT DROP (markMissingMessagesFromLiveUidStream) instead of passing a bigint[] parameter — needed for Gmail-scale folders (100k+ UIDs).
  • Advisory lock is session-level + explicit unlock (src/locks.ts). The test repository-safety.test.ts > "does not hold account locks with idle transactions" pins this against accidental conversion to pg_advisory_xact_lock.
  • MIME parser is hardened. Body-structure visitors have a depth cap (64) + WeakSet cycle guard; simpleParser is called with maxHtmlLengthToParse: 1 MB and link/text skip flags; decodeHtmlEntities rejects out-of-range and surrogate code points to prevent a single email DoSing the worker.
  • API error contract is structured. app.onError maps ZodError → 400, HostValidationError → 400, NotFoundError → 404, pg 23505 → 409; everything else logs structured JSON and returns { error: \"internal_error\" }.

What's deferred (explicit non-goals for v0.1)

  • camelCase vs snake_case on API responses (input is camelCase, response is DB snake_case).
  • Library hook error semantics: a thrown hook currently aborts the sync run; intentional for v0.1.
  • `htmlToText` is still a multi-pass regex (single-pass tokenizer is a future perf win).
  • `fetchBodyBacklog` is still per-message (batched fetch is a follow-up).

Test plan

  • `pnpm install` (lockfile is current).
  • `pnpm typecheck` — clean.
  • `pnpm test` — 59 passing across 10 files, including new behavioral suites for crypto round-trip, host validation, MIME edge cases (out-of-range entities, cyclic body structure, deep nesting).
  • `pnpm build` — clean.
  • (Optional, requires local Supabase) `supabase db start && supabase db reset --local && pnpm dry-run:local` — covers schema apply, account create, two syncs, hook firing, idempotency.
  • (Optional, requires Docker) `pnpm smoke:greenmail` — covers real-protocol sync against a disposable IMAP server.

🤖 Generated with Claude Code

fedster99 and others added 11 commits May 3, 2026 22:17
- Encrypt IMAP passwords in Node (AES-256-GCM, versioned payload)
  and drop the pgcrypto pgp_sym_encrypt/decrypt SQL helpers so the
  master key never crosses into Postgres logs or pg_stat_statements
- Validate IMAP host/port before connect: reject private/link-local/
  loopback/metadata IPs, restrict ports to 143/993, require TLS unless
  IMAP_ALLOW_PRIVATE_HOSTS opt-in is set
- Constant-time bearer comparison; structured API error contract
  (zod input validation, global onError mapping ZodError/host-validation/
  not-found/pg unique-violation to 400/404/409); separate ADMIN_TOKEN
  for POST /migrate
- Harden MIME parsing: depth + cycle guards on bodyStructure visitors,
  surrogate / out-of-range code-point rejection in entity decoder,
  simpleParser size + link-skip limits
- Sanitize sync_state_reason / sync-run error text against credential
  echo and control-char log injection at every persistence site
- Schema: bigint size_bytes/raw_bytes, supporting indexes for the
  ON DELETE SET NULL FKs and for message_id_normalized lookups,
  folder_path documented as the canonical key
- Stream reconcile UIDs through a TEMP TABLE ON COMMIT DROP so heavy
  mailboxes (100k+ UIDs) don't OOM the worker
- Worker: uncaughtException / unhandledRejection handlers feed
  graceful shutdown so the advisory lock is released cleanly;
  SIGTERM aborts the in-flight sync between accounts
- Wrap each message + attachment upsert in its own transaction so
  a partial attachment insert can't leave a half-formed row
- Recovery: a clean sync clears DEGRADED/BROKEN back to HEALTHY
- Cleanup: replace Math.max(...uids) spread, recursive throttle
  acquire, dead hooks.ts module, unused ImapThrottle observability
- Tests: behavioral suites for crypto round-trip, host validation
  (IPv4/IPv6 reserved ranges, ports, TLS requirement), and MIME
  edge cases (out-of-range entities, cyclic / deeply nested
  bodyStructure). 59 tests across 10 files.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant