Implement AI rate limiting, bot automation, and user settings page

.github/SECRETS.md

@@ -1,113 +1,58 @@
 # GitHub Secrets Configuration
 
-This document lists all the secrets that need to be configured in GitHub repository settings for CI/CD pipelines to work properly.
+All secrets needed for CI/CD pipelines. Configure in **Settings → Secrets and variables → Actions**.
 
 ## Required Secrets
 
-### Docker Hub (for docker-build job)
+### VPS Deployment
 
-- **DOCKER_USERNAME** - Your Docker Hub username
-- **DOCKER_PASSWORD** - Your Docker Hub password or access token
+| Secret        | Description                         | Example       |
+| ------------- | ----------------------------------- | ------------- |
+| `VPS_HOST`    | VPS IP address or hostname          | `203.0.113.1` |
+| `VPS_USER`    | SSH username on VPS                 | `deploy`      |
+| `VPS_PORT`    | SSH port                            | `22`          |
+| `VPS_SSH_KEY` | Private SSH key for the deploy user | Full PEM key  |
 
-### Code Coverage (optional)
+### Container Registry (GHCR)
 
-- **CODECOV_TOKEN** - Codecov token for uploading coverage reports
-  - Get from: https://codecov.io/
-
-### Deployment Secrets
-
-#### Staging Environment
-
-- **STAGING_DATABASE_URL** - PostgreSQL connection string for staging
-  - Format: `postgresql://user:password@host:port/database`
-
-#### Production Environment
-
-- **PRODUCTION_DATABASE_URL** - PostgreSQL connection string for production
-  - Format: `postgresql://user:password@host:port/database`
-
-### Notifications (optional)
-
-- **SLACK_WEBHOOK_URL** - Slack webhook URL for deployment notifications
-  - Create at: https://api.slack.com/messaging/webhooks
+> [!NOTE]
+> GHCR uses `GITHUB_TOKEN` automatically — no additional secrets needed for pushing images.
 
 ## How to Add Secrets
 
 1. Go to your GitHub repository
-2. Click on **Settings** → **Secrets and variables** → **Actions**
+2. Click **Settings** → **Secrets and variables** → **Actions**
 3. Click **New repository secret**
-4. Add each secret with its corresponding value
+4. Add each secret listed above
 
-## Environment-Specific Secrets
+## VPS Setup Checklist
 
-### Staging
+Before the deploy workflow can succeed, ensure the VPS has:
 
-Go to **Settings** → **Environments** → **staging** → **Add secret**
+1. Docker and Docker Compose installed
+2. The `deploy` user with docker group access
+3. Project directory at `/opt/postly` with `.env` file (`chmod 600`)
+4. GHCR login configured: `docker login ghcr.io -u <github-user> -p <PAT>`
+5. SSH key added to `~/.ssh/authorized_keys` for the deploy user
 
-### Production
-
-Go to **Settings** → **Environments** → **production** → **Add secret**
-
-## Security Best Practices
-
-1. **Never commit secrets to the repository**
-2. **Rotate secrets regularly** (every 3-6 months)
-3. **Use environment-specific secrets** for staging vs production
-4. **Limit secret access** to specific workflows/environments
-5. **Enable branch protection** for main and develop branches
-6. **Require approval** for production deployments
-
-## Additional Configuration
-
-### Branch Protection Rules (Recommended)
+## Branch Protection Rules (Recommended)
 
 For `main` branch:
 
 - ✅ Require pull request reviews before merging
-- ✅ Require status checks to pass before merging
-  - lint
-  - type-check
-  - test
-  - build
+- ✅ Require status checks to pass (lint, type-check, test, build)
 - ✅ Require branches to be up to date before merging
-- ✅ Require conversation resolution before merging
 - ✅ Do not allow bypassing the above settings
 
-For `develop` branch:
-
-- ✅ Require status checks to pass before merging
-- ✅ Require branches to be up to date before merging
-
-### Environment Protection Rules
-
-For `production` environment:
-
-- ✅ Required reviewers (at least 1)
-- ✅ Wait timer: 5 minutes
-- ✅ Deployment branches: Only `main` and tags matching `v*`
-
-For `staging` environment:
-
-- ✅ Deployment branches: Only `main` and `develop`
-
-## Verifying Configuration
-
-After adding secrets, you can verify they're working by:
-
-1. Pushing to a feature branch
-2. Creating a pull request to `develop` or `main`
-3. Check that CI workflow runs successfully
-4. For deployment, merge to `main` and verify deployment workflow
-
-## Troubleshooting
-
-If workflows fail due to missing secrets:
-
-1. Check workflow logs for specific error messages
-2. Verify secret names match exactly (case-sensitive)
-3. Ensure secrets are set in the correct environment
-4. Check that workflow has permission to access the secret
+## Rollback
 
-## Contact
+Every deploy tags images with the Git SHA. To rollback:
 
-For questions about secrets configuration, contact your DevOps team or repository administrator.
+```bash
+ssh deploy@<VPS_HOST>
+cd /opt/postly
+export API_IMAGE=ghcr.io/<repo>/api:<previous-sha>
+export SCRAPER_IMAGE=ghcr.io/<repo>/scraper:<previous-sha>
+export BOT_IMAGE=ghcr.io/<repo>/bot:<previous-sha>
+docker compose -f docker-compose.prod.yml up -d --no-deps api bot scraper
+```

.gitignore

@@ -108,4 +108,5 @@ lerna-debug.log*
 test-results/
 playwright-report/
 blob-report/
-playwright/.cache/
+playwright/.cache/
+apps/scraper/.playwright_data

README.md

@@ -0,0 +1,182 @@
+# Postly — Production Deployment Guide
+
+Complete runbook for deploying Postly to a VPS.
+
+---
+
+## Architecture Overview
+
+```
+VPS Proxy (Nginx/Traefik) → API (Express)
+                          → Static Web (Vite build)
+
+Internal Docker Network:
+  API ←→ PostgreSQL (pgvector)
+  API ←→ Redis (BullMQ + Caching)
+  Scraper ←→ PostgreSQL
+  Bot ←→ PostgreSQL + Redis
+```
+
+**Stack:** Node.js API · Python Scraper · Python Discord Bot · PostgreSQL 16 + pgvector · Redis 7
+
+---
+
+## Quick-Start Checklist
+
+```
+□  1. Clone repo to /opt/postly, create .env
+□  2. docker compose -f docker-compose.prod.yml up -d
+□  3. Verify all services healthy
+□  4. Configure GitHub Actions secrets
+□  5. Push to main → verify pipeline runs
+□  6. Set up cron backup job
+□  7. Run a backup restore drill
+```
+
+---
+
+## Step-by-Step Deployment
+
+### 1. Clone and Configure
+
+Log into your VPS and run:
+
+```bash
+cd /opt/postly
+git clone https://github.com/<your-repo>.git .
+
+# Create production .env from template
+cp .env.production.example .env
+chmod 600 .env
+
+# Edit .env — fill in all CHANGE_ME values
+nano .env
+```
+
+**Critical .env values to set:**
+
+- `DB_PASSWORD` — Strong random password (`openssl rand -hex 16`)
+- `JWT_SECRET` / `JWT_REFRESH_SECRET` — (`openssl rand -hex 32`)
+- `DISCORD_BOT_TOKEN` — From Discord Developer Portal
+- `WEB_URL` — Your production domain (e.g., `https://postly.io`)
+
+### 2. Login to GHCR
+
+```bash
+# Login to pull pre-built images from GitHub Container Registry
+echo "<YOUR_PAT>" | docker login ghcr.io -u <github-username> --password-stdin
+```
+
+### 3. Start the Stack
+
+```bash
+docker compose -f docker-compose.prod.yml up -d
+```
+
+Verify all services are healthy:
+
+```bash
+docker compose -f docker-compose.prod.yml ps
+curl -s http://localhost:3000/health | jq
+```
+
+Expected health response:
+
+```json
+{
+  "status": "ok",
+  "checks": { "db": "ok", "redis": "ok" },
+  "uptime": 12.345
+}
+```
+
+### 4. Run HNSW Index Migration (One-time)
+
+```bash
+docker exec -i postly-postgres psql -U postly -d postly < scripts/add-hnsw-indexes.sql
+```
+
+### 5. Setup Backups
+
+```bash
+# Make backup script executable
+chmod +x scripts/backup.sh
+
+# Test it manually first
+bash scripts/backup.sh
+
+# Add to cron (runs daily at 2 AM)
+(crontab -l 2>/dev/null; echo "0 2 * * * /opt/postly/scripts/backup.sh >> /var/log/postly-backup.log 2>&1") | crontab -
+```
+
+### 6. Configure GitHub Actions
+
+Add these secrets in **Settings → Secrets → Actions**:
+
+| Secret        | Value                      |
+| ------------- | -------------------------- |
+| `VPS_HOST`    | Your VPS IP address        |
+| `VPS_USER`    | `deploy`                   |
+| `VPS_PORT`    | `22`                       |
+| `VPS_SSH_KEY` | Full private SSH key (PEM) |
+
+Push to `main` and verify the pipeline deploys successfully.
+
+---
+
+## Rollback
+
+Every deploy tags images with the Git SHA. To rollback:
+
+```bash
+cd /opt/postly
+export API_IMAGE=ghcr.io/<repo>/api:<previous-sha>
+export SCRAPER_IMAGE=ghcr.io/<repo>/scraper:<previous-sha>
+export BOT_IMAGE=ghcr.io/<repo>/bot:<previous-sha>
+docker compose -f docker-compose.prod.yml up -d --no-deps api bot scraper
+```
+
+---
+
+## Backup Restore Drill
+
+Run this monthly to verify backups work:
+
+```bash
+# Start a throwaway Postgres container
+docker run -d --name pg-restore-test -e POSTGRES_PASSWORD=test pgvector/pgvector:pg16
+
+# Restore latest backup into it
+docker exec -i pg-restore-test pg_restore -U postgres -d postgres --create < backups/local/$(ls -t backups/local/ | head -1)
+
+# Verify data
+docker exec pg-restore-test psql -U postgres -d postly -c "SELECT count(*) FROM users;"
+
+# Clean up
+docker rm -f pg-restore-test
+```
+
+---
+
+## Scaling Roadmap
+
+| Users   | Action                                          | Cost Impact |
+| ------- | ----------------------------------------------- | ----------- |
+| 0–1K    | Current setup, no changes                       | —           |
+| 1K–10K  | Add Postgres read replica (second VPS)          | +€4.5/mo    |
+| 10K–50K | Extract scraper to own VPS, add pgBouncer       | +€4.5/mo    |
+| 50K+    | Consider managed DB, split into domain services | Variable    |
+
+---
+
+## File Reference
+
+| File                           | Purpose                          |
+| ------------------------------ | -------------------------------- |
+| `docker-compose.prod.yml`      | Main production stack            |
+| `scripts/backup.sh`            | Daily PostgreSQL backup          |
+| `scripts/add-hnsw-indexes.sql` | pgvector HNSW indexes (run once) |
+| `.env.production.example`      | Production env template          |
+| `.github/workflows/deploy.yml` | CI/CD pipeline                   |
+| `.github/workflows/ci.yml`     | PR checks                        |
+| `.github/SECRETS.md`           | GitHub secrets reference         |

apps/api/package.json

@@ -16,8 +16,9 @@
     "@dodopayments/express": "^0.2.6",
     "@postly/config": "*",
     "@postly/database": "*",
+    "@postly/logger": "*",
     "@postly/shared-types": "*",
-    "bcryptjs": "^2.4.3",
+    "bcrypt": "^5.1.1",
     "bullmq": "^5.31.3",
     "cors": "^2.8.6",
     "dotenv": "^16.4.7",
@@ -29,12 +30,13 @@
     "mammoth": "^1.11.0",
     "multer": "^2.0.2",
     "pdf-parse": "^2.4.5",
+    "resend": "^6.9.4",
     "zod": "^3.24.1"
   },
   "devDependencies": {
     "@postly/eslint-config": "*",
     "@postly/typescript-config": "*",
-    "@types/bcryptjs": "^2.4.6",
+    "@types/bcrypt": "^5.0.2",
     "@types/cors": "^2.8.17",
     "@types/express": "^5.0.0",
     "@types/jsonwebtoken": "^9.0.7",

apps/api/src/config/secrets.ts

@@ -34,4 +34,6 @@ export {
   DODO_PAYMENTS_WEBHOOK_KEY,
   DODO_PAYMENTS_ENVIRONMENT,
   DODO_PAYMENTS_RETURN_URL,
+  RESEND_API_KEY,
+  RESEND_FROM_EMAIL,
 } from "@postly/config";