Release staging to main

.agents/skills/databuddy-internal/SKILL.md

@@ -21,6 +21,7 @@ Keep additions **minimal**: one bullet, a new `rg` hint, or a routing note—eno
 - Never use production/customer data as tests, fixtures, snapshots, examples, or copied output. Tests must use placeholders/mocks only (example.com, example IDs). If production ClickHouse is queried for investigation, summarize anonymized aggregates and do not paste customer domains, client IDs, emails, or other identifiers into code or responses.
 - `apps/dashboard`: Next.js app on port `3000` (per-website **agent** chat: `@ai-sdk/react` `useChat` via `contexts/chat-context.tsx` — not the separate `chat-sdk` package; overlapping sends while streaming are queued client-side to mirror a “queue latest” strategy.)
 - Dashboard Playwright webServer commands run under CI PATH from setup-bun; avoid `bash -lc` because login shells can drop Bun from PATH. Build dist-only workspace packages such as `@databuddy/sdk` and `@databuddy/devtools` before starting the API/dashboard. Client `NEXT_PUBLIC_*` flags must use direct env access so Next can inline them. `readBooleanEnv` only treats the literal string `"true"` as enabled, so CI E2E booleans must use `"true"`/`"false"`, not `"1"`/`"0"`.
+- Dashboard Playwright public/demo analytics specs call API `/v1/query` anonymously from the browser; keep `DATABUDDY_E2E_MODE` query behavior isolated from production rate limits so CI retries do not exhaust `anon:unknown`.
 - `apps/api`: Elysia API on port `3001`
 - `apps/slack`: Slack agent adapter; Slack installs must resolve through org-scoped DB integration records, not a single env bot token/default website. Agent calls must use an encrypted per-integration Databuddy API key secret as a normal bearer token, never a global internal secret.
 - Slack OAuth lives in `apps/api`, but slash commands/events require `apps/slack` to be running too; local `bun run dev:dashboard` runs dashboard + API only, so use `bun run dev:slack` when working on Slack. The Slack package scripts read the root `.env`.
@@ -30,10 +31,12 @@ Keep additions **minimal**: one bullet, a new `rg` hint, or a routing note—eno
 - Slack memory is separate from billing/auth: pass a Slack-scoped `memoryUserId` such as `slack-{team}-{user}` plus current-speaker context so one Slack user's saved name/preferences do not bleed into another user's replies.
 - Slack agent write tools need the integration automation API key to include the matching Databuddy API scopes (currently `read:data`, `read:links`, `write:links`, `manage:websites`, `manage:flags`); older installs may need reconnecting so a new key is minted.
 - Shared agent integrations should call `@databuddy/ai/agent` (`askDatabuddyAgent` / `streamDatabuddyAgent`) instead of importing internal MCP run/history helpers directly.
+- Insights generation logic belongs in `apps/insights` and should reuse `@databuddy/ai`; `apps/api` should only read insight data or queue runs, not own prompts, model calls, tool loops, validation, or persistence orchestration.
 - Agent ClickHouse SQL must use the canonical analytics.events schema: `client_id`, `time`, `path`, `event_name`, and pageviews as `event_name = 'screen_view'`; never `website_id`, `created_at`, `page_path`, `event_type`, or `pageview`.
 - Slack agent evals live in `packages/evals`: use `bun run eval --surface slack` for the whole Slack surface. `--tag slack` is only a tiny smoke subset, and `cost_fallback` in agent telemetry is pricing-catalog fallback, not proof the model request fell back.
 - Slack agent expected stops such as exhausted Databunny credits should throw `DatabuddyAgentUserError` from `@databuddy/ai/agent/errors`; Slack surfaces those messages directly and reserves the generic reconnect copy for real infrastructure failures.
 - Slack Docker builds use `bun build --compile --bytecode`; keep `apps/slack/src/index.ts` bootstrapping inside an async `main()` instead of top-level `await`, which can fail during compile even when typecheck passes.
+- Insights Docker builds also use `bun build --compile --bytecode`; keep `apps/insights/src/index.ts` startup work inside async functions instead of top-level `await`.
 - After Slack Docker changes, verify the full pruned image with `docker build --progress=plain -f slack.Dockerfile -t databuddy-slack:test .`; the inner Bun compile is not enough because prune can miss dependency build outputs and package exports.
 - Slack-reachable shared packages (`@databuddy/ai`, `@databuddy/rpc`) must not import `evlog/elysia`; use host-injected request logger providers from the API and plain evlog fallbacks elsewhere.
 - AI link tools must assign link folders by existing folder `id` or `slug` only; folder names are display text and must not be used for routing or dedupe.
@@ -98,6 +101,8 @@ Read [codebase-map.md](./references/codebase-map.md) when you need deeper routin
 - Insights merged feed (`use-insights-feed`) collapses history + AI by `insightSignalDedupeKey` in `apps/dashboard/lib/insight-signal-key.ts` so the list is one row per signal (latest wins).
 - Insights page (`app/(main)/insights`) should stay focused on the brief + signal queue; do not add generic global analytics KPI cards or top pages/referrers/countries tables there.
 - Theme: `apps/dashboard/app/globals.css`. **`--border` is intentionally subtle**; do not crank it darker for “contrast” unless **iza** asks—prefer text tokens or layout for readability.
+- Website analytics filters are two-way synced between Jotai and the `filters` URL param in `app/(main)/websites/[id]/layout.tsx`; guard URL-driven atom writes from echoing stale atom state back into `nuqs`, or adding a filter can lock the page during form submit.
+- Do not centralize, relocate, or otherwise refactor dashboard E2E API route access gates during cleanup; keep test-only access checks local to each route unless iza explicitly asks for that change.
 - Integration catalog logos: use filled Simple Icons SVG path data (or equivalent filled brand SVG), store the path on each item as `iconPath`, render it through a shared logo tile with `bg-secondary/60`, `border-border/70`, `text-foreground`, and `fill="currentColor"`, then use brand color only as a small accent bar (`accent` or `accentClassName: "bg-foreground/70"` for black/near-black brands). Avoid raw brand-black icons or mixed line/filled icon sets that disappear in dark mode.
 - Organization integrations settings should stay list-first and operational: coming-soon integrations are static rows, Slack is the only expandable row for now, and connected integrations need obvious lifecycle controls such as uninstall/disconnect in the row details.
 - Dashboard UI must use `apps/dashboard/components/ds` primitives exactly; feature code must not use raw form/control elements (`button`, `input`, `select`, `textarea`, native dialogs), Base UI/Radix primitives, or ad hoc styled controls directly. If a variant is missing, add or extend the DS component first. For menu-style folder/status/filter/sort/action pickers, use `components/ds/dropdown-menu.tsx`; use `Select` only when the established pattern is explicitly a select/combobox. Read `apps/dashboard/components/ds/README.md` before creating new dashboard UI.

.env.example

@@ -5,8 +5,18 @@ DATABASE_URL="postgres://databuddy:databuddy_dev_password@localhost:5432/databud
 DB_POOL_MAX="10"
 REDIS_URL="redis://localhost:6379"
 BULLMQ_REDIS_URL="redis://localhost:6379"
+# Optional dedicated BullMQ Redis URL for the insights worker. Falls back to BULLMQ_REDIS_URL.
+INSIGHTS_PORT="4002"
+INSIGHTS_BULLMQ_REDIS_URL=""
+INSIGHTS_DISPATCH_INTERVAL_MS="300000"
+INSIGHTS_MAINTENANCE_INTERVAL_MS="300000"
+INSIGHTS_STALE_ITEM_MS="900000"
+INSIGHTS_WORKER_CONCURRENCY="5"
+INSIGHTS_WORKER_ENABLED="true"
+INSIGHTS_EVLOG_FS=""
 
 AI_GATEWAY_API_KEY=""
+SUPERMEMORY_API_KEY=""
 
 BETTER_AUTH_URL="http://localhost:3000"
 BETTER_AUTH_SECRET="generate-a-random-32-byte-base64-secret"

.github/workflows/ci.yml

@@ -136,6 +136,12 @@ jobs:
         env:
           NODE_ENV: test
         run: bun run test
+      - name: Insights integration
+        env:
+          NODE_ENV: test
+          INSIGHTS_INTEGRATION_TESTS: "true"
+          BULLMQ_REDIS_URL: redis://localhost:6379/4
+        run: bun run --cwd apps/insights test:integration
       - name: Uptime router integration
         env:
           NODE_ENV: test

.github/workflows/docker-publish.yml

@@ -49,7 +49,7 @@ jobs:
           EVENT_NAME: ${{ github.event_name }}
           BEFORE_SHA: ${{ github.event.before }}
         run: |
-          ALL='["api","basket","dashboard","links","uptime"]'
+          ALL='["api","basket","dashboard","insights","links","uptime"]'
           if [[ "$EVENT_NAME" != "push" ]]; then
             echo "services=$ALL" >> "$GITHUB_OUTPUT"
             exit 0
@@ -61,7 +61,7 @@ jobs:
           export TURBO_SCM_BASE="$BEFORE_SHA"
           export TURBO_SCM_HEAD="HEAD"
           affected=()
-          for svc in api basket dashboard links uptime; do
+          for svc in api basket dashboard insights links uptime; do
             count=$(bunx turbo ls --affected --filter="@databuddy/$svc" --output=json | jq -r '.packages.count')
             if [[ "$count" != "0" ]]; then
               affected+=("\"$svc\"")
@@ -103,6 +103,8 @@ jobs:
             description: "Databuddy Basket service - event ingestion"
           - service: dashboard
             description: "Databuddy Dashboard service - web analytics UI"
+          - service: insights
+            description: "Databuddy Insights service - queued insight generation"
           - service: links
             description: "Databuddy Links service - URL shortening and tracking"
           - service: uptime

.github/workflows/health-check.yml

@@ -8,6 +8,7 @@ on:
       - ".dockerignore"
       - "apps/api/**"
       - "apps/basket/**"
+      - "apps/insights/**"
       - "packages/**"
       - "bun.lock"
       - "package.json"
@@ -20,6 +21,7 @@ on:
       - ".dockerignore"
       - "apps/api/**"
       - "apps/basket/**"
+      - "apps/insights/**"
       - "packages/**"
       - "bun.lock"
       - "package.json"
@@ -267,3 +269,115 @@ jobs:
           fi
 
           echo "Basket health check passed!"
+
+  insights-health-check:
+    name: Insights Health Check
+    runs-on: blacksmith-4vcpu-ubuntu-2404
+    timeout-minutes: 20
+
+    services:
+      redis:
+        image: redis:7-alpine
+        ports:
+          - 6379:6379
+        options: >-
+          --health-cmd "redis-cli ping"
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+      postgres:
+        image: postgres:17-alpine
+        env:
+          POSTGRES_USER: postgres
+          POSTGRES_PASSWORD: postgres
+          POSTGRES_DB: databuddy_test
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd "pg_isready -U postgres -d databuddy_test"
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
+
+      - name: Mount Docker build cache
+        uses: useblacksmith/stickydisk@41873b1513bb679f9c115504cbd13d3660432504 # v1
+        with:
+          key: ${{ github.repository }}-docker-build-cache
+          path: /tmp/docker-build-cache
+
+      - name: Set up Docker Builder
+        uses: useblacksmith/setup-docker-builder@ac083cc84672d01c60d5e8561d0a939b697de542 # v1
+
+      - name: Build Insights Docker image
+        uses: useblacksmith/build-push-action@cbd1f60d194a98cb3be5523b15134501eaf0fbf3 # v2
+        with:
+          context: .
+          file: ./insights.Dockerfile
+          push: false
+          load: true
+          tags: insights:test
+
+      - name: Run Insights health check
+        run: |
+          set -euo pipefail
+          trap 'docker rm -f insights-health-check >/dev/null 2>&1 || true' EXIT
+
+          docker run -d \
+            --name insights-health-check \
+            --network host \
+            -e NODE_ENV=test \
+            -e PORT=4002 \
+            -e DATABASE_URL=postgresql://postgres:postgres@localhost:5432/databuddy_test \
+            -e REDIS_URL=redis://localhost:6379 \
+            -e BULLMQ_REDIS_URL=redis://localhost:6379/4 \
+            -e INSIGHTS_BULLMQ_REDIS_URL= \
+            -e INSIGHTS_DISPATCH_INTERVAL_MS=60000 \
+            -e INSIGHTS_MAINTENANCE_INTERVAL_MS=60000 \
+            -e INSIGHTS_STALE_ITEM_MS=300000 \
+            -e INSIGHTS_WORKER_CONCURRENCY=1 \
+            -e INSIGHTS_WORKER_ENABLED=true \
+            -e BETTER_AUTH_SECRET=test-better-auth-secret-for-health-checks \
+            -e AI_GATEWAY_API_KEY=test-ai-gateway-key \
+            -e SUPERMEMORY_API_KEY= \
+            insights:test
+
+          echo "Waiting for Insights to start..."
+          for i in {1..30}; do
+            if curl -sf http://localhost:4002/health > /dev/null 2>&1; then
+              echo "Insights is responding!"
+              break
+            fi
+            if [ $i -eq 30 ]; then
+              echo "Insights failed to start within 30 seconds"
+              docker logs insights-health-check
+              exit 1
+            fi
+            sleep 1
+          done
+
+          STATUS_BODY=$(curl -sS http://localhost:4002/health/status)
+          echo "Insights /health/status: $STATUS_BODY"
+          if echo "$STATUS_BODY" | jq -e '.status == "ok"' > /dev/null; then
+            echo "Insights dependency health is valid"
+          else
+            echo "Insights dependency health is not ok"
+            docker logs insights-health-check
+            exit 1
+          fi
+
+          RESPONSE=$(curl -sf http://localhost:4002/health || echo '{}')
+          echo "Insights /health: $RESPONSE"
+
+          if echo "$RESPONSE" | jq -e '.workerEnabled == true' > /dev/null; then
+            echo "Insights health endpoint structure is valid"
+          else
+            echo "Insights health endpoint response missing expected workerEnabled=true"
+            docker logs insights-health-check
+            exit 1
+          fi
+
+          echo "Insights health check passed!"

apps/api/src/index.ts

@@ -21,7 +21,6 @@ import {
 import { openApiHandler } from "@/rpc/openapi";
 import { agent } from "./routes/agent";
 import { health } from "./routes/health";
-import { insights } from "./routes/insights";
 import { integrations } from "./routes/integrations";
 import { mcp } from "./routes/mcp";
 import { publicApi } from "./routes/public";
@@ -90,7 +89,6 @@ const app = new Elysia({ precompile: true })
 	.use(query)
 	.use(agent)
 	.use(integrations)
-	.use(insights)
 	.use(mcp)
 	.all("/rpc/*", handleRpcEndpoint, { parse: "none" })
 	.all("/", handleOpenApiReference, { parse: "none" })