No More ML and Sqlite

.coveragerc

@@ -0,0 +1,21 @@
+[run]
+source = website_profiling
+
+# We are NOT hiding core code. We only omit modules that require external
+# services/binaries (Google APIs, Lighthouse) or are impractical to unit-test here.
+omit =
+    */website_profiling/integrations/google/*
+    */website_profiling/lighthouse/*
+    */website_profiling/reporting/*
+    */website_profiling/tools/*
+    */website_profiling/security_scanner.py
+    */website_profiling/llm/providers/*
+    */website_profiling/llm/*
+    */website_profiling/llm_config.py
+    */website_profiling/cli.py
+    */website_profiling/commands/enrich_cmd.py
+
+[report]
+show_missing = True
+skip_empty = True
+

.github/workflows/ci.yml

@@ -0,0 +1,58 @@
+name: CI
+
+on:
+  push:
+    branches: [main, master]
+  pull_request:
+
+jobs:
+  python:
+    runs-on: ubuntu-latest
+    services:
+      postgres:
+        image: postgres:16-alpine
+        env:
+          POSTGRES_USER: profiling
+          POSTGRES_PASSWORD: profiling
+          POSTGRES_DB: website_profiling
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd "pg_isready -U profiling -d website_profiling"
+          --health-interval 5s
+          --health-timeout 3s
+          --health-retries 5
+    env:
+      DATABASE_URL: postgres://profiling:profiling@localhost:5432/website_profiling
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+      - name: Install dependencies
+        run: pip install -r requirements.txt
+      - name: Apply migrations
+        run: alembic upgrade head
+      - name: Pytest
+        run: pytest tests/ -q
+
+  web:
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: web
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '20'
+          cache: npm
+          cache-dependency-path: web/package-lock.json
+      - name: Install
+        run: npm ci
+      - name: Typecheck
+        run: npm run typecheck
+      - name: Lint
+        run: npm run lint
+      - name: Test
+        run: npm test

.gitignore

@@ -2,19 +2,29 @@
 
 # Next.js UI: generated pipeline configs from the runner modal (repo root; must match Python cwd for paths)
 .website-profiling-ui-*.txt
-# Legacy: configs under subfolder broke sqlite_db = report.db vs Next (kept for ignore if present)
+# Legacy UI pipeline configs under subfolder (kept for ignore if present)
 .web-pipeline/
 
-# WebsiteProfiling generated outputs
+# Local runtime data (shadow config, secrets volume mount)
+data/*
+!data/.gitkeep
+
+# Python virtual environments (README recommends .venv; ignore both names at repo root)
+.venv/
+venv/
+
+# Legacy SQLite outputs (if present locally)
 report.db
 report.db.*
-
 report.db-journal
 nodes.json
 
 # Google integration secrets (never commit)
 .secrets/
 web/.env.local
 
-# Legacy local config file (use report.db / web UI instead)
+# Legacy local config file (use PostgreSQL pipeline_config / web UI instead)
 input.txt
+# Runtime shadow written to DATA_DIR by the web UI (not the committed example)
+pipeline-config.txt
+*__pycache__*

AGENT.md

@@ -1,34 +1,41 @@
 # Agent instructions -- WebsiteProfiling
 
-**What it is:** `python -m src` from repo root (`src/__main__.py` -> package **`website_profiling`**). Config: stored in **`report.db`** (`pipeline_config` table, `key/value/is_unknown/updated_at`). A shadow **`pipeline-config.txt`** is auto-written next to `report.db` on every Save/Run. CLI loads DB first (`REPORT_DB_PATH` or `cwd/report.db`), then shadow file; `--config` overrides with a file. Reference keys: `input.txt.example` (not auto-loaded).
+**What it is:** `python -m src` from repo root (`src/__main__.py` -> package **`website_profiling`**). Config: stored in **PostgreSQL** (`pipeline_config` table, `key/value/is_unknown/updated_at`). A shadow **`pipeline-config.txt`** is auto-written to `DATA_DIR` on every Save/Run. CLI loads DB first (`DATABASE_URL`), then shadow file; `--config` overrides with a file. Reference keys: `input.txt.example` and `pipeline-config.example.txt` (not auto-loaded).
 
-**Frontend:** **`web/`** (Next.js) -- server reads `report.db` via `/api/report/*`.
+**LLM / AI:** Settings live in **`llm_config`** table in PostgreSQL. Configure only via web UI **AI** tab (`GET/PUT /api/llm-config`, localhost). Never in `pipeline-config.txt` or `--config`.
+
+**Frontend:** **`web/`** (Next.js) -- server reads PostgreSQL via `/api/report/*`.
 
 **Key paths**
 
-- `src/website_profiling/` -- `cli.py`, `config.py`, `crawl/`, `db/storage.py`, `lighthouse/`, `reporting/`, `ml/enrich.py`, `tools/`
-- `web/app/` -- routes; `web/src/` -- React; pipeline: `PipelineRunnerFab`, `server/pipelineJobs.js`, `server/pipelineConfig.js`
+- `src/website_profiling/` -- `cli.py`, `config.py`, `crawl/`, `db/storage.py`, `lighthouse/`, `reporting/`, `analysis/`, `llm/`, `tools/`
+- `web/app/` -- routes; `web/src/` -- React; pipeline: `PipelineRunnerFab`, `server/pipelineJobs.ts`, `server/pipelineConfig.ts`, `server/llmConfig.ts`, `server/db.ts`
+- `alembic/` -- schema migrations
 
 **Run / APIs**
 
-- Pipeline: `python -m src` — reads config from `report.db` (`pipeline_config`); shadow `pipeline-config.txt` if table empty. CLI override: `python -m src --config path`
-- Optional step: `crawl` | `report` | `plot` | `lighthouse` | `keywords` | `warnings` | `enrich`
-- **`preserve_crawl_history`** (default true): append crawls; `false` recreates crawl tables but restores `report_payload`, Lighthouse, `google_data`, `keyword_data`, `keyword_history`, `keyword_suggest_cache`, and `crawl_runs`
-- **`enrich_keywords_after_report`**: when omitted or `auto` (UI: Auto), follows `enable_google_search_console`; when set to Yes/No, explicit override
-- **`REPORT_DB_PATH`** env: DB path used by both Python and Next.js (Docker: `/data/report.db`; local default: `report.db` at repo root). Pipeline config lives in this DB.
-- **`web/`:** `/api/report/*` (SQLite); `/api/run` spawns Python (localhost only); `/api/pipeline-config` GET/PUT for persistent settings; `PipelineRunnerFab` saves state to `report.db` (`pipeline_config` table) + shadow `pipeline-config.txt` before each run
-- **Docker:** `Dockerfile` + `docker-compose.yml`; **`LIGHTHOUSE_CHROME_FLAGS`**; ML caches under `/data/cache/*` in compose
+- Pipeline: `python -m src` — reads config from PostgreSQL (`pipeline_config`); shadow `DATA_DIR/pipeline-config.txt` if table empty. CLI override: `python -m src --config path`
+- Optional step: `crawl` | `report` | `plot` | `lighthouse` | `keywords` | `warnings` | `enrich` | `google`
+- **`preserve_crawl_history`** (default true): append crawls; `false` truncates crawl tables but restores `report_payload`, Lighthouse, `google_data`, `keyword_data`, `keyword_history`, `keyword_suggest_cache`, and `crawl_runs`
+- **`DATABASE_URL`** env: PostgreSQL connection string (required). **`DATA_DIR`**: secrets + shadow config (Docker: `/data`).
+- **Pipeline data** (crawl, edges, nodes, report payload, Lighthouse, keywords, warnings) is stored in **PostgreSQL only** — no JSON/CSV/HTML exports from the main pipeline.
+- **Pool tuning:** `DB_POOL_MIN` / `DB_POOL_MAX` (Python), `PGPOOL_MAX` (Node). Bulk crawl writes via `executemany`; optional **`crawl_stream_to_db`** streams rows during fetch.
+- **`web/`:** `/api/report/*` (PostgreSQL); `/api/run` spawns Python (localhost only); `/api/pipeline-config` GET/PUT; `/api/llm-config` GET/PUT (AI only); `PipelineRunnerFab` saves pipeline + LLM state before each run
+- **Job store:** in-memory on `globalThis` in `web/src/server/pipelineJobs.ts` — job status/log is lost on server restart (single-process dev/Docker only).
+- **Docker:** `Dockerfile` + `docker-compose.yml` (postgres + web); **`LIGHTHOUSE_CHROME_FLAGS`**
 
 **Where to edit**
 
 | Task | Where |
 |------|--------|
 | Crawl | `crawl/crawler.py` |
 | Report | `reporting/builder.py`, `reporting/categories.py` |
-| DB schema | `db/storage.py` `init_schema` |
-| ML | `ml/enrich.py`, `requirements-ml.txt` |
+| DB schema | `alembic/versions/` |
+| Local analysis | `analysis/local.py`, `requirements.txt` |
+| LLM enrichment | `llm/enrich.py`, `llm_config.py`, `requirements-llm.txt` |
 | Config / CLI | `config.py` (`load_config`, `load_config_from_db`), `cli.py`, `input.txt.example` |
-| UI config schema | `web/src/lib/pipelineConfigSchema.js` |
-| UI config I/O | `web/src/server/pipelineConfig.js` |
+| UI pipeline schema | `web/src/lib/pipelineConfigSchema.ts` |
+| UI LLM schema | `web/src/lib/llmConfigSchema.ts` |
+| UI config I/O | `web/src/server/pipelineConfig.ts`, `web/src/server/llmConfig.ts` |
 
-Schema changes: edit `init_schema` only (no migration layer). ML stack: prefer Python **3.12** for spaCy/blis; **3.13** may fail pip builds.
+Schema changes: add Alembic migration (`alembic revision`).

Dockerfile

@@ -5,7 +5,7 @@
 
 FROM node:20-bookworm-slim AS base
 
-# Python venv + Chromium + build tools (ML stack may compile native wheels on some platforms)
+# Python venv + Chromium + build tools
 RUN apt-get update && apt-get install -y --no-install-recommends \
     build-essential \
     python3 \
@@ -34,22 +34,30 @@ ENV PYTHONDONTWRITEBYTECODE=1 \
     PYTHONUNBUFFERED=1 \
     NEXT_TELEMETRY_DISABLED=1 \
     WEBSITE_PROFILING_ROOT=/app \
-    REPORT_DB_PATH=/data/report.db \
+    DATABASE_URL=postgres://profiling:profiling@postgres:5432/website_profiling \
+    DATA_DIR=/data \
     PYTHON=/opt/venv/bin/python \
-    CHROME_PATH=/usr/bin/chromium
+    CHROME_PATH=/usr/bin/chromium \
+    LIGHTHOUSE_PATH=/usr/local/bin/lighthouse
 
-# Python: base requirements, then ML/NLP (sentence-transformers, spaCy, KeyBERT, scikit-learn, …)
+# Python: base requirements + optional LLM API clients
 COPY requirements.txt /app/requirements.txt
-COPY requirements-ml.txt /app/requirements-ml.txt
+COPY requirements-llm.txt /app/requirements-llm.txt
+COPY alembic.ini /app/alembic.ini
+COPY alembic /app/alembic
 RUN --mount=type=cache,target=/root/.cache/pip \
     python3 -m venv /opt/venv \
   && /opt/venv/bin/pip install --upgrade pip \
   && /opt/venv/bin/pip install -r /app/requirements.txt \
-  && /opt/venv/bin/pip install -r /app/requirements-ml.txt \
-  && /opt/venv/bin/python -m spacy download en_core_web_sm \
+  && /opt/venv/bin/pip install -r /app/requirements-llm.txt \
   && ln -sf /opt/venv/bin/python /usr/local/bin/python \
   && ln -sf /opt/venv/bin/python /usr/local/bin/python3
 
+# Pre-install Lighthouse CLI (avoid flaky parallel `npx -y lighthouse` at runtime).
+RUN --mount=type=cache,target=/root/.npm \
+    npm install -g lighthouse@12.6.0 \
+  && lighthouse --version
+
 WORKDIR /app
 
 # Next.js install + build (layer cache)
@@ -60,15 +68,17 @@ RUN --mount=type=cache,target=/root/.npm \
 # Application source
 COPY src /app/src
 COPY web /app/web
+COPY alembic /app/alembic
+COPY alembic.ini /app/alembic.ini
+COPY docker-entrypoint.sh /app/docker-entrypoint.sh
 
 RUN cd /app/web && npm run build && npm prune --omit=dev
 
 ENV NODE_ENV=production
 
-# Persisted DB directory (bind mount or volume in compose)
-RUN mkdir -p /data
+# Persisted data directory (secrets + shadow config)
+RUN mkdir -p /data && chmod +x /app/docker-entrypoint.sh
 
 EXPOSE 3000
 
-# Listen on all interfaces so the container is reachable from the host
-CMD ["sh", "-c", "cd /app/web && npm run start -- -H 0.0.0.0 -p 3000"]
+CMD ["/app/docker-entrypoint.sh"]

Docs.md

@@ -1,6 +1,8 @@
 # Crawl overview
 
-**What it does:** Starts from `start_url` in **pipeline config** (`report.db` / web UI), fetches HTML with HTTP GET, parses `<a href>` with BeautifulSoup (static HTML only—no JS execution), normalizes links, filters (same-site, robots, depth, excludes), and queues until `max_pages` or the queue is empty.
+**Config load order (CLI):** `--config` file → PostgreSQL `pipeline_config` → shadow `DATA_DIR/pipeline-config.txt` (see `src/website_profiling/cli.py`).
+
+**What it does:** Starts from `start_url` in **pipeline config** (PostgreSQL / web UI), fetches HTML with HTTP GET, parses `<a href>` with BeautifulSoup (static HTML only—no JS execution), normalizes links, filters (same-site, robots, depth, excludes), and queues until `max_pages` or the queue is empty.
 
 **Main code:** `src/website_profiling/crawl/crawler.py`, `src/website_profiling/common.py`.
 

README.md

@@ -8,11 +8,40 @@ From the **repository root**:
 docker compose up --build
 ```
 
-Open **http://localhost:3000/home**. Use **`http://localhost:3000`**
+Open **http://localhost:3000/home** (the site root `http://localhost:3000` redirects to `/home`).
+
+Docker Compose starts **PostgreSQL** and the web app. Data persists in Docker volumes (`pg-data` for the database, `profiling-data` for secrets and shadow config).
+
+### PostgreSQL credentials
+
+| Environment | `DATABASE_URL` |
+|-------------|----------------|
+| Docker Compose | `postgres://profiling:profiling@postgres:5432/website_profiling` (set in `docker-compose.yml`) |
+| Local dev (example below) | `postgres://postgres:dev@localhost:5432/website_profiling` |
 
 ## Run locally
 
-**1. Python** (repo root)
+**1. PostgreSQL**
+
+```bash
+docker run -d --name wp-pg \
+  -e POSTGRES_PASSWORD=dev \
+  -e POSTGRES_DB=website_profiling \
+  -p 5432:5432 postgres:16-alpine
+
+export DATABASE_URL=postgres://postgres:dev@localhost:5432/website_profiling
+export DATA_DIR=$(pwd)/data
+mkdir -p "$DATA_DIR"
+```
+
+Apply schema:
+
+```bash
+pip install -r requirements.txt
+alembic upgrade head
+```
+
+**2. Python** (repo root)
 
 ```bash
 python -m venv .venv
@@ -24,36 +53,67 @@ Activate `.venv`, then:
 pip install -r requirements.txt
 ```
 
-Optional ML: `pip install -r requirements-ml.txt`
+Optional LLM enrichment: `pip install -r requirements-llm.txt` — configure in the web UI **AI** tab only.
 
-**2. Configure & run the pipeline**
+**3. Configure & run the pipeline**
 
 The easiest way is via the **web UI** (terminal icon, bottom-right corner at `http://localhost:3000`):
-- Settings are stored in `report.db` (`pipeline_config` table) — the same database used for crawl data. Back up the whole pipeline by copying one file.
-- A shadow `pipeline-config.txt` is auto-written next to `report.db` on every Save/Run (safe to delete; regenerated automatically).
+- Settings are stored in PostgreSQL (`pipeline_config` table).
+- A shadow `pipeline-config.txt` is auto-written to `DATA_DIR` on every Save/Run.
 - On first open, if the table is empty, the UI imports from shadow `pipeline-config.txt` (if present).
-- Click **Save settings** to persist, or **Run pipeline** to save + run immediately.
+- **AI enrichment**: use the **AI** tab — settings live in `llm_config` in PostgreSQL only.
 
-To run from the CLI instead:
+To run from the CLI:
 
 ```bash
+export DATABASE_URL=postgres://postgres:dev@localhost:5432/website_profiling
 python -m src
 ```
 
-Python reads settings from `report.db` (`pipeline_config` table) by default — use the web UI to configure, or set `REPORT_DB_PATH` to point at your database (Docker sets this automatically). If the table is empty, the CLI falls back to shadow `pipeline-config.txt` next to `report.db`. Override with `--config path` for a custom key=value file. Steps: `crawl`, `report`, `plot`, `lighthouse`, `keywords`, `warnings`, `enrich` as extra args.
-
-> **Reference:** `input.txt.example` shows all config keys in the legacy file format (optional; not loaded automatically).
-
-**3. Next.js UI** (`web/`)
+**4. Next.js UI** (`web/`)
 
 ```bash
 cd web
 npm install
+export DATABASE_URL=postgres://postgres:dev@localhost:5432/website_profiling
+export DATA_DIR=../data
 npm run dev
 ```
 
 Open **http://localhost:3000/home**.
 
+If pipeline runs fail with `spawn python ENOENT`, macOS often has no `python` on PATH (only `python3`). The server auto-resolves `.venv/bin/python` or `python3` (see `web/src/server/resolvePython.ts`). You can also:
+
+- Set `export PYTHON="$(pwd)/.venv/bin/python"` before `npm run dev`, or
+- Set **Python executable** under **Pipeline → Settings → Advanced** (persisted in the browser), or
+- Set optional **Repo root** there if the app cannot find `src/__main__.py`.
+
+Pipeline job status is in-memory only and is cleared when the Next.js server restarts.
+
+### PostgreSQL performance tuning
+
+Optional environment variables:
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `DB_POOL_MIN` | 2 | Python pipeline minimum pool connections |
+| `DB_POOL_MAX` | 20 | Python pipeline maximum pool connections |
+| `PGPOOL_MAX` | 20 | Next.js `pg` pool size |
+
+Pipeline config (web UI or shadow file):
+
+- **`crawl_stream_to_db`** — batch-write crawl rows during fetch (auto-enabled when `max_pages > 100`).
+- **`lighthouse_concurrency`** — parallel Lighthouse URL audits (default 2).
+- **`llm_concurrency`** (AI tab) — parallel LLM API batches (default 2).
+
+Benchmark crawl writes: `python scripts/bench_crawl_write.py -n 1000` (requires `DATABASE_URL`).
+
+### Backup
+
+```bash
+pg_dump -Fc "$DATABASE_URL" -f backup.dump
+```
+
 ---
 
 ## Google Search Console + GA4 Integration
@@ -80,7 +140,7 @@ Pull real search and traffic data into your reports.
 ### CLI usage
 
 ```bash
-# Fetch GSC + GA4 data and store in report.db (uses pipeline_config in report.db)
+# Fetch GSC + GA4 data and store in PostgreSQL
 python -m src google
 
 # Validate credentials only (does not store data)