feat: PageIndex-style corpus tools + MCP server + deploy hardening

.claude/skills/cloudflare-harden/SKILL.md

@@ -0,0 +1,229 @@
+---
+name: cloudflare-harden
+description: End-to-end Cloudflare hardening playbook for a self-hosted app on a single VPS reverse-proxied by Caddy. Takes a bootstrapped box from "Let's Encrypt + wide-open firewall + public site" to "Cloudflare Origin CA + Full strict TLS + proxied DNS + UFW locked to Cloudflare IPs + Zero Trust Access". Use when the user says "harden the box", "lock origin to cloudflare", "add cloudflare access", "ufw + cloudflare", "cloudflare proxy + access setup", or describes putting Cloudflare in front of a self-hosted origin. Requires the cloudflare-api MCP server (or a CF API token) and ssh root@ to the origin.
+---
+
+# Cloudflare hardening playbook
+
+The exact sequence applied to `from-zero.emeraldlake.io` (Hetzner CX23, Caddy → Next.js). Run from a laptop that has both the Cloudflare MCP connected and SSH access to the origin.
+
+## Parameters to collect from the user before starting
+
+| Var | Example | How to derive |
+|---|---|---|
+| `HOSTNAME` | `from-zero.emeraldlake.io` | the FQDN to harden |
+| `ROOT_DOMAIN` | `emeraldlake.io` | parent zone, for the CF zone lookup |
+| `ORIGIN_IP` | `178.105.102.91` | server's public IP (already in DNS if grey-cloud is set) |
+| `APP_PORT` | `3717` | what the reverse_proxy target is |
+| `ALLOWED_EMAILS` | `tarek.kekhia@emeraldlake.io` | one or more for the Access policy |
+| `SESSION_DURATION` | `720h` | CF Access cookie lifetime |
+| `ZONE_ID` | `1a8035c3...` | resolved via `GET /zones?name=<ROOT_DOMAIN>` |
+| `ACCOUNT_ID` | preset by MCP | exposed as `accountId` in mcp__cloudflare-api__execute |
+
+Ask the user only for the ones that aren't obvious. Use `mcp__cloudflare-api__execute` to look up `ZONE_ID` from `ROOT_DOMAIN`. Confirm `HOSTNAME` and `ALLOWED_EMAILS` explicitly.
+
+## Pre-flight
+
+The origin must already be running Caddy + the app behind it on `127.0.0.1:APP_PORT`. If the box was bootstrapped by `deploy/bootstrap.sh`, Caddy is installed but no Caddyfile is in place yet (that's intentional: we install the TLS-enabled Caddyfile as part of this playbook, after the Origin CA cert exists).
+
+Verify before proceeding:
+
+```bash
+ssh root@<ORIGIN_IP> 'systemctl is-active caddy; ss -tlnp | grep -E ":80|:443|:<APP_PORT>"; ls /etc/caddy/tls/ 2>/dev/null'
+```
+
+## The 8 steps
+
+### 1. DNS A record (grey cloud, temporary)
+
+```js
+// in cloudflare-api MCP
+await cloudflare.request({ method: "POST", path: `/zones/${ZONE_ID}/dns_records`, body: {
+  type: "A", name: HOSTNAME, content: ORIGIN_IP, proxied: false, ttl: 1,
+}});
+```
+
+Verify: `dig +short <HOSTNAME> @1.1.1.1` returns `ORIGIN_IP`.
+
+### 2. Generate Origin CA cert
+
+Run on **the laptop** (so the private key is generated locally, never round-tripped through Cloudflare):
+
+```bash
+WORK=/tmp/cf-origin-${HOSTNAME}
+mkdir -p "$WORK" && cd "$WORK"
+openssl req -new -newkey rsa:2048 -nodes -keyout origin.key -out origin.csr \
+  -subj "/CN=${HOSTNAME}"
+chmod 600 origin.key
+```
+
+POST the CSR to Cloudflare via MCP. Capture the returned cert:
+
+```js
+const csr = await fs.readFile(`${WORK}/origin.csr`, "utf8");
+const r = await cloudflare.request({ method: "POST", path: "/certificates", body: {
+  hostnames: [HOSTNAME],
+  request_type: "origin-rsa",
+  requested_validity: 5475,  // 15 years
+  csr,
+}});
+// save r.result.certificate to ${WORK}/origin.pem
+```
+
+**Critical: do NOT use the `Write` tool to save the cert file.** Use `cat > origin.pem <<'CERT' ... CERT` via Bash. Reason: during the first attempt for this project, something clobbered the local private key with the cert content right after Write was called on the cert path. We lost the key and had to revoke + reissue. Always verify with:
+
+```bash
+openssl x509 -in origin.pem -pubkey -noout | openssl md5
+openssl pkey -in origin.key -pubout 2>/dev/null | openssl md5
+# the two hashes MUST match
+```
+
+Save the returned `id` from the API response so you can revoke later if needed.
+
+### 3. Install cert + Caddyfile on origin
+
+Use `deploy/install-origin-cert.sh` — it scps the pair and the project Caddyfile, places them with correct ownership/perms, validates Caddyfile, reloads Caddy.
+
+```bash
+bash deploy/install-origin-cert.sh /tmp/cf-origin-${HOSTNAME}/origin.pem /tmp/cf-origin-${HOSTNAME}/origin.key ${ORIGIN_IP}
+```
+
+Verify Caddy logs are clean (look for `tls.cache.maintenance` lines, ignore the benign `no OCSP stapling for [cloudflare origin certificate]: no URL to issuing certificate` warning — Origin CA certs don't carry OCSP URLs).
+
+Sanity-check the origin is serving the new cert:
+
+```bash
+echo | openssl s_client -connect ${ORIGIN_IP}:443 -servername ${HOSTNAME} 2>/dev/null \
+  | openssl x509 -noout -issuer -dates
+# issuer should be: CloudFlare Origin SSL Certificate Authority
+```
+
+### 4. Cloudflare SSL/TLS mode → Full (strict)
+
+```js
+await cloudflare.request({ method: "PATCH", path: `/zones/${ZONE_ID}/settings/ssl`,
+  body: { value: "strict" }});
+```
+
+### 5. Flip DNS to proxied (orange cloud)
+
+Find the DNS record id, then PATCH it:
+
+```js
+const dns = await cloudflare.request({ method: "GET", path: `/zones/${ZONE_ID}/dns_records`, query: { name: HOSTNAME } });
+const dnsId = dns.result[0].id;
+await cloudflare.request({ method: "PATCH", path: `/zones/${ZONE_ID}/dns_records/${dnsId}`, body: { proxied: true } });
+```
+
+### 6. Verify edge
+
+```bash
+CF_IP=$(dig +short ${HOSTNAME} @1.1.1.1 | head -1)
+curl -sS -o /dev/null -D - --resolve ${HOSTNAME}:443:${CF_IP} https://${HOSTNAME}/ \
+  | grep -iE '^(http|server|cf-ray):'
+# expect: HTTP/2 200, server: cloudflare, cf-ray: ...
+```
+
+Confirm visitors see a publicly-trusted edge cert (issuer will be Google Trust Services or similar — Cloudflare's edge cert provider). The Origin CA cert is now only used between Cloudflare and your box.
+
+### 7. Lock UFW to Cloudflare IPs
+
+Already in the repo as `deploy/lock-origin-to-cloudflare.sh`. SCP and run:
+
+```bash
+scp deploy/lock-origin-to-cloudflare.sh root@${ORIGIN_IP}:/tmp/
+ssh root@${ORIGIN_IP} bash /tmp/lock-origin-to-cloudflare.sh
+```
+
+Verify direct origin hits time out:
+
+```bash
+curl -sS -o /dev/null -k --resolve ${HOSTNAME}:443:${ORIGIN_IP} \
+  -w "Direct: HTTP %{http_code} in %{time_total}s\n" --max-time 10 \
+  https://${HOSTNAME}/ 2>&1
+# expect: timeout / errno 28 / 10s
+```
+
+### 8. Cloudflare Access (optional gate)
+
+Use the **zone-scoped** Access endpoints — the project's MCP token has zone Access scope but not account-level Access scope:
+
+```js
+// Create the app
+const app = await cloudflare.request({ method: "POST",
+  path: `/zones/${ZONE_ID}/access/apps`,
+  body: {
+    type: "self_hosted",
+    name: `Access gate for ${HOSTNAME}`,
+    domain: HOSTNAME,
+    session_duration: SESSION_DURATION,
+    auto_redirect_to_identity: false,
+    app_launcher_visible: true,
+    allowed_idps: [],   // empty = all available, including the built-in One-time PIN
+  },
+});
+const APP_ID = app.result.id;
+
+// Create the allow policy
+await cloudflare.request({ method: "POST",
+  path: `/zones/${ZONE_ID}/access/apps/${APP_ID}/policies`,
+  body: {
+    name: "Allowlist",
+    decision: "allow",
+    include: ALLOWED_EMAILS.map(email => ({ email: { email } })),
+    precedence: 1,
+  },
+});
+```
+
+Verify: unauthenticated request returns 302 to `<team>.cloudflareaccess.com/cdn-cgi/access/login/...`.
+
+```bash
+curl -sS -D - -o /dev/null --max-redirs 0 --resolve ${HOSTNAME}:443:${CF_IP} \
+  https://${HOSTNAME}/ | head -5
+# expect: HTTP/2 302 with location header pointing to cloudflareaccess.com
+```
+
+## Pre-existing requirements / setup steps
+
+**Zero Trust enablement** is a one-time, dashboard-only step. If `GET /zones/${ZONE_ID}/access/apps` returns `9999: access.api.error.not_enabled`, tell the user: open https://one.dash.cloudflare.com/ → pick a team subdomain (becomes `<team>.cloudflareaccess.com`) → choose Free plan. Then re-try. No API path exists to enable Zero Trust for the first time.
+
+## Verification checklist (run at the very end)
+
+```bash
+dig +short ${HOSTNAME} @1.1.1.1                                    # CF anycast IP
+curl -I https://${HOSTNAME}                                        # 302 to <team>.cloudflareaccess.com (or 200 if no Access)
+curl -k --resolve ${HOSTNAME}:443:${ORIGIN_IP} https://${HOSTNAME}/ # timeout (UFW dropping)
+ssh root@${ORIGIN_IP} 'ufw status verbose | head -25'              # 80,443 only from CF ranges, 22 open
+ssh root@${ORIGIN_IP} 'echo | openssl s_client -connect 127.0.0.1:443 -servername '${HOSTNAME}' 2>/dev/null | openssl x509 -noout -issuer'  # CF Origin CA
+```
+
+## Rollback recipes
+
+| To undo | Run |
+|---|---|
+| Access gate | `DELETE /zones/${ZONE_ID}/access/apps/${APP_ID}` |
+| UFW lockdown | restore wide-open: `ufw allow 80,443/tcp` then `ufw reload`, plus delete the CF rules (script supports `rm /var/lib/cloudflare-ufw.list` + manual rules cleanup) |
+| Cloudflare proxy | PATCH dns_record `proxied: false` (grey cloud) |
+| Strict SSL | PATCH `/zones/${ZONE_ID}/settings/ssl` to `"full"` or `"flexible"` |
+| Origin CA cert | DELETE `/certificates/${CERT_ID}` (Cloudflare revokes; the operator still needs to swap the Caddyfile back to ACME and reload Caddy) |
+
+## Things that look scary but aren't
+
+- Caddy logs `no OCSP stapling for [cloudflare origin certificate]: no URL to issuing certificate` — Origin CA certs don't have OCSP URLs. Benign.
+- During step 5, my local resolver may have cached NXDOMAIN. Bypass with `dig @1.1.1.1` or `curl --resolve`.
+- `/user/tokens/verify` returns "Invalid API Token" through the MCP — the MCP uses a non-standard auth path; ignore.
+
+## Things to actually fear
+
+- **Losing the Origin CA private key.** It's never sent to Cloudflare. If lost, the cert is unusable. Revoke + reissue. Always pubkey-match cert + key right after generation.
+- **Flipping the proxy on with Let's Encrypt + ACME still active.** Caddy will silently fail renewals (CF intercepts :80 for HTTP-01) and the cert eventually expires. Solution is what this playbook does: replace ACME with the Origin CA cert before flipping orange-cloud.
+- **UFW-lockdown before proxy is on.** Will brick the site for everyone including the operator (Caddy can't be reached). Always do step 7 AFTER step 6 verifies the edge route works.
+
+## Related files in this repo
+
+- `deploy/install-origin-cert.sh` — origin-side cert + Caddyfile installer (idempotent)
+- `deploy/lock-origin-to-cloudflare.sh` — UFW lockdown (idempotent, has state file)
+- `deploy/Caddyfile` — the TLS-enabled Caddyfile that points at `/etc/caddy/tls/origin.{pem,key}`
+- `deploy/bootstrap.sh` — fresh-box bootstrap; intentionally does NOT install the Caddyfile or start Caddy
+- `deploy/README.md` — runbook that references this skill

.gitignore

@@ -23,6 +23,16 @@ web/data/*.db-shm
 zero/_pdf-output/
 zero/_pdf-tooling/node_modules/
 
+# Per-environment user-data tracking files. The seed templates live next
+# to them as *.example.md and are committed; the live files are gitignored
+# and auto-copied from the templates on first run by
+# web/src/lib/content-loader.ts → bootstrapUserData(). Treat them like
+# .env vs .env.local.
+zero/04-learning/glossary.md
+zero/04-learning/knowledge-tracker.md
+zero/04-learning/questions-and-answers.md
+zero/05-meta/progress-log.md
+
 # python
 __pycache__/
 *.pyc

deploy/Caddyfile

@@ -1,12 +1,9 @@
-{
-  email tarek.kekhia@emeraldlake.io
-}
-
 from-zero.emeraldlake.io {
+  tls /etc/caddy/tls/origin.pem /etc/caddy/tls/origin.key
+
   encode zstd gzip
   reverse_proxy 127.0.0.1:3717
 
-  # Sensible defaults
   request_body {
     max_size 5MB
   }

deploy/README.md

@@ -9,12 +9,19 @@ deploying on every push to `main`.
 
 | File                  | Purpose                                                    |
 |-----------------------|------------------------------------------------------------|
-| `bootstrap.sh`        | One-shot system bootstrap. Run as root on a fresh server.  |
+| `bootstrap.sh`        | One-shot system bootstrap. Run as root on a fresh server. Leaves Caddy on default config; TLS arrives via the harden playbook. |
 | `deploy.sh`           | Idempotent deploy. Run as `lithium` user. Called by GHA.   |
+| `install-origin-cert.sh` | Installs a Cloudflare Origin CA cert + the project Caddyfile on the box. Verifies cert/key pair match first. |
+| `lock-origin-to-cloudflare.sh` | Replaces wide-open ufw 80/443 rules with allow-lists scoped to Cloudflare's published IP ranges. |
 | `lithium.service`     | systemd unit for the Next.js process.                      |
-| `Caddyfile`           | Reverse proxy + auto-TLS for the domain.                   |
+| `Caddyfile`           | Reverse proxy + Cloudflare Origin CA TLS for the domain.   |
 | `lithium.env.example` | Template for `/etc/lithium/lithium.env` (secrets live here).|
 
+For the end-to-end Cloudflare hardening sequence (Origin CA + Full strict +
+orange-cloud + UFW lockdown + Zero Trust Access), see the project skill at
+[`.claude/skills/cloudflare-harden/SKILL.md`](../.claude/skills/cloudflare-harden/SKILL.md).
+In Claude Code, say "harden the box" to invoke it.
+
 ## Architecture
 
 ```
@@ -32,10 +39,11 @@ during deploy never touches the running app's data.
 
 ## Bring-up — one time
 
-### 1. DNS
+### 1. DNS (grey cloud, temporary)
 
 Point an `A` record for `from-zero.emeraldlake.io` at the Hetzner box's
-public IP. Caddy fetches a Let's Encrypt cert on first request.
+public IP. Start in **DNS-only (grey cloud)** mode — the hardening
+playbook flips it to proxied later, once the Origin CA cert is in place.
 
 ### 2. System bootstrap
 
@@ -51,7 +59,10 @@ ssh root@<server-ip> bash /tmp/deploy/bootstrap.sh
 
 This installs Node 20, Caddy, build tools, creates the `lithium` user,
 state dirs, swap, ufw rules, sudoers entry, and drops the systemd unit
-and Caddyfile into place.
+in place. It intentionally leaves Caddy on its default config — the
+project Caddyfile references an Origin CA cert that doesn't exist yet,
+so installing the Caddyfile here would crash Caddy. TLS arrives via the
+harden playbook in step 7.
 
 ### 3. Generate the deploy SSH key (on the server)
 
@@ -100,7 +111,24 @@ Watch logs:
 sudo journalctl -u lithium -f
 ```
 
-Visit <https://from-zero.emeraldlake.io>.
+The app is now listening on `127.0.0.1:3717` but not reachable from the
+internet yet — Caddy is still on its default config.
+
+### 7. Cloudflare hardening
+
+Open this repo in Claude Code and say **"harden the box"**, or follow
+[`.claude/skills/cloudflare-harden/SKILL.md`](../.claude/skills/cloudflare-harden/SKILL.md)
+manually. The playbook walks through:
+
+1. Generate Origin CA cert (RSA 2048, valid 15 years) via Cloudflare MCP.
+2. Install cert + Caddyfile on the box (`deploy/install-origin-cert.sh`).
+3. Set Cloudflare SSL/TLS mode to **Full (strict)**.
+4. Flip DNS to **proxied (orange cloud)**.
+5. Lock UFW to Cloudflare IP ranges only (`deploy/lock-origin-to-cloudflare.sh`).
+6. (Optional) Add a **Cloudflare Access** application + policy so visitors
+   must authenticate before any request reaches the origin.
+
+After this, visit <https://from-zero.emeraldlake.io>.
 
 ## Day-2
 

deploy/bootstrap.sh

@@ -108,9 +108,14 @@ cp "$(dirname "$0")/lithium.service" /etc/systemd/system/lithium.service
 systemctl daemon-reload
 systemctl enable lithium.service
 
-echo "==> Installing Caddyfile"
-cp "$(dirname "$0")/Caddyfile" /etc/caddy/Caddyfile
-systemctl restart caddy
+echo "==> Caddy left at its default config"
+# The project Caddyfile uses /etc/caddy/tls/origin.{pem,key} (Cloudflare
+# Origin CA cert). Those files don't exist on a fresh box, so installing the
+# Caddyfile here would crash Caddy. Operator installs cert + Caddyfile
+# together via deploy/install-origin-cert.sh as part of the
+# `.claude/skills/cloudflare-harden` playbook. Caddy stays on its default
+# config until then.
+systemctl enable caddy >/dev/null 2>&1 || true
 
 echo "==> Seeding env file at ${ENV_FILE} (chmod 600, owner ${APP_USER})"
 if [ ! -f "${ENV_FILE}" ]; then
@@ -126,15 +131,17 @@ Bootstrap complete.
 
 Next steps:
 
-1. Point DNS A-record  ${DOMAIN}  →  this server's IP. Caddy will fetch a
-   Let's Encrypt cert automatically on the next request.
+1. Point DNS A-record  ${DOMAIN}  →  this server's IP.
+   Start in DNS-only (grey cloud) mode; the cloudflare-harden playbook
+   flips it to proxied later.
 
 2. Generate a deploy SSH key on this box (as root or as ${APP_USER}):
 
      sudo -u ${APP_USER} ssh-keygen -t ed25519 -f /home/${APP_USER}/.ssh/id_ed25519 -N ''
      sudo -u ${APP_USER} cat /home/${APP_USER}/.ssh/id_ed25519.pub >> /home/${APP_USER}/.ssh/authorized_keys
+     chown ${APP_USER}:${APP_USER} /home/${APP_USER}/.ssh/authorized_keys
      sudo -u ${APP_USER} chmod 600 /home/${APP_USER}/.ssh/authorized_keys
-     sudo -u ${APP_USER} cat /home/${APP_USER}/.ssh/id_ed25519       # private — paste into GH secret SSH_KEY
+     sudo -u ${APP_USER} cat /home/${APP_USER}/.ssh/id_ed25519       # paste into GH secret SSH_KEY
 
 3. Add these GitHub repository secrets at
    https://github.com/emeraldtarek/from-zero/settings/secrets/actions:
@@ -155,7 +162,18 @@ Next steps:
        bash deploy/deploy.sh
      '
 
-6. From now on, every push to main triggers GH Actions → SSH → deploy.sh.
+   App listens on 127.0.0.1:3717 but isn't reachable yet — Caddy is still
+   on its default config.
+
+6. Harden with Cloudflare. In Claude Code, say "harden the box" or run the
+   playbook in .claude/skills/cloudflare-harden/SKILL.md. It walks through:
+     - Origin CA cert generation + install via deploy/install-origin-cert.sh
+     - Setting SSL/TLS mode to Full (strict)
+     - Flipping DNS to proxied
+     - Locking UFW to Cloudflare IPs via deploy/lock-origin-to-cloudflare.sh
+     - (optional) Adding a Cloudflare Access gate
+
+7. From now on, every push to main triggers GH Actions → SSH → deploy.sh.
 
 App will be live at:  https://${DOMAIN}
 ==============================================================================