Skip to content

Add SupaMail OSS IMAP-to-Supabase core#1

Draft
fedster99 wants to merge 10 commits into
mainfrom
codex/oss-worker-core
Draft

Add SupaMail OSS IMAP-to-Supabase core#1
fedster99 wants to merge 10 commits into
mainfrom
codex/oss-worker-core

Conversation

@fedster99
Copy link
Copy Markdown
Owner

@fedster99 fedster99 commented May 4, 2026

What changed

  • Frames the spun-out project as SupaMail: reliable IMAP sync for Supabase.
  • Adds the standalone TypeScript core for syncing IMAP mailboxes into Supabase/Postgres.
  • Adds neutral mirror tables for accounts, folders, messages, raw/parsed message bodies, attachment metadata, sync runs, and sync events.
  • Adds an integrated worker with folder discovery, metadata sync, UIDVALIDITY reset handling, reconciliation, advisory-lock serialization, body fetching, and health/backoff state.
  • Stores full message bodies through the core engine: raw RFC822/MIME plus parsed text, HTML, headers, MIME structure, parser metadata, and attachment metadata.
  • Adds body fetch policies: immediate, lazy, and priority_then_backfill.
  • Adds generic IMAP and Rackspace provider profiles.
  • Adds a Hono API, CLI, Dockerfile, Fly.io worker/API configs, deployment docs, schema docs, local dry run, smoke tests, and GitHub Actions CI.
  • Rewrites the README around the SupaMail name, background, Supabase + Fly.io setup, query examples, and product positioning.

Review hardening

  • Requires API_TOKEN for the API service and keeps only /health public.
  • Returns sanitized account summaries from API/CLI-facing account paths instead of encrypted password rows or lock internals.
  • Enables RLS on mirror tables, revokes anon/authenticated table access when those Supabase roles exist, and revokes public execution on password helper functions.
  • Reclaims stale currently_syncing leases using heartbeat age so crashed workers do not brick an account.
  • Treats partial folder failures as degraded sync health instead of marking the account healthy.
  • Reconciles against all live provider UIDs and guards against empty UID results on non-empty mailboxes.
  • Deduplicates attachment metadata by stable MIME part number.
  • Avoids immediately untracking folders after one missing LIST pass.
  • Cleans remaining imap-to-supabase runtime naming drift.

Deployment shape

  • Fly.io is the hosted deployment path: one small always-on worker with no public IP, plus an optional separate API app.
  • The Fly starter commands use --ha=false so the cheapest first deploy stays to one Machine.
  • Docker Compose/Coolify remains documented as the self-hosted/VPS path.

Known follow-ups

Why

This turns the Signal IMAP sync engine into a separate OSS/self-hosted project instead of landing a large invasive extraction PR inside Signal.

SupaMail exists for teams and builders whose inboxes are not neatly covered by Gmail/Outlook-first tooling. The goal is simple: connect any IMAP mailbox, mirror email into Supabase reliably, and make it easy to build AI agents, internal tools, CRM workflows, search, alerts, and automations on top of owned database tables.

How it works

SupaMail treats IMAP as the provider and Supabase/Postgres as the durable mailbox mirror. The worker serializes account operations with Postgres advisory locks, tracks folder cursors and UIDVALIDITY, fetches metadata and full bodies according to policy, and runs reconciliation so missing messages do not silently become permanent gaps.

The default operational shape is Supabase + Fly.io: Supabase hosts Postgres, Fly runs the always-on worker without a public IP, and the optional API can run as a separate Fly app.

Validation

  • pnpm typecheck
  • pnpm test
  • pnpm build
  • DATABASE_URL=postgresql://postgres:postgres@localhost:54322/postgres IMAP_ENCRYPTION_KEY=... pnpm dry-run:local
  • Fresh temporary local Postgres database dry-run through scripts/local-dry-run.ts

Notes

  • This is intentionally email sync only: no calendar, contacts, sending, scheduling, CRM, or AI logic in the core.
  • Signal-specific CRM hydration, identity/belief pipelines, MCP routes, Trigger.dev coupling, and dashboard code are excluded.
  • Supabase transaction-pooler URLs are rejected because account advisory locks require session affinity.
  • The PR stays draft while SupaMail is dogfooded and the public surface settles.

@fedster99 fedster99 changed the title [codex] Add OSS IMAP mirror core Add SupaMail OSS IMAP-to-Supabase core May 5, 2026
@fedster99
Copy link
Copy Markdown
Owner Author

Review follow-up from the local fedster99/check-pr-issue-status workspace:

I addressed the pre-landing review findings locally. These changes are not visible in this PR until they are pushed onto the PR branch.

Fixed locally:

  • Sanitized sync errors before DB persistence, worker logs, API responses, and hook exposure.
  • Hardened IMAP host validation for IPv4-mapped IPv6 private/reserved addresses, including compressed and expanded hex forms.
  • Allowed private-host dev ports so the GreenMail smoke setup can use its forwarded local IMAP port.
  • Made successful syncs recover unhealthy accounts back to HEALTHY.
  • Stopped excluding Archive folders as noisy mail.
  • Reworked account advisory locks to avoid long idle transactions.
  • Reworked full reconcile to stream live UIDs into a temp table in batches instead of sending one huge SQL array.

Verification passed locally:

  • pnpm typecheck
  • pnpm test (10 files, 59 tests)
  • pnpm build
  • Targeted host-validation checks for mapped IPv6 blocking and GreenMail port allowance

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant