Mcp and chat

.github/workflows/ci.yml

@@ -39,15 +39,17 @@ jobs:
         run: |
           pytest tests/test_categories_roadmap.py tests/test_report_categories_golden.py \
             tests/test_categories_coverage.py tests/test_indexation_coverage.py tests/test_crawl_segments.py \
-            tests/test_terminology.py \
+            tests/test_terminology.py tests/test_compare_payload.py \
             --cov=website_profiling.reporting --cov-config=.coveragerc.reporting \
             --cov-report=term-missing --cov-fail-under=100 -q -o addopts=
       - name: Pytest (tools coverage gate)
         run: |
           pytest tests/test_alert_checker.py tests/test_schedule_runner.py tests/test_export_audit.py \
-            tests/test_export_audit_coverage.py \
+            tests/test_export_audit_coverage.py tests/test_audit_tools.py tests/test_audit_tools_expanded.py \
+            tests/test_audit_tools_coverage.py \
+            tests/test_mcp_registry.py tests/test_mcp_resources.py \
             --cov=website_profiling.tools --cov-config=.coveragerc.tools \
-            --cov-report=term-missing --cov-fail-under=100 -q -o addopts=
+            --cov-report=term-missing --cov-fail-under=95 -q -o addopts=
       - name: CLI smoke
         run: python -m src --help
 

AGENT.md

@@ -19,12 +19,14 @@
 **Run / APIs**
 
 - Run audit (CLI): `python -m src` — reads config from PostgreSQL (`pipeline_config`); shadow `DATA_DIR/pipeline-config.txt` if table empty. CLI override: `python -m src --config path`
-- Optional step: `crawl` | `report` | `plot` | `lighthouse` | `keywords` | `warnings` | `enrich` | `google`
+- Optional step: `crawl` | `report` | `plot` | `lighthouse` | `keywords` | `warnings` | `enrich` | `google` | `chat`
 - **`preserve_crawl_history`** (default true): append crawls; `false` truncates crawl tables but restores `report_payload`, Lighthouse, `google_data`, `keyword_data`, `keyword_history`, `keyword_suggest_cache`, and `crawl_runs`
 - **`DATABASE_URL`** env: PostgreSQL connection string (required). **`DATA_DIR`**: secrets + shadow config (Docker: `/data`).
 - **Pipeline data** (crawl, edges, nodes, report payload, Lighthouse, keywords, warnings) is stored in **PostgreSQL only** — no JSON/CSV/HTML exports from the main pipeline.
 - **Pool tuning:** `DB_POOL_MIN` / `DB_POOL_MAX` (Python), `PGPOOL_MAX` (Node). Bulk crawl writes via `executemany`; optional **`crawl_stream_to_db`** streams rows during fetch.
-- **`web/`:** `/api/report/*` (PostgreSQL); `/api/run` spawns Python (localhost only); `/api/crawl/browser-status` GET (localhost, Playwright/Chromium preflight); `/api/pipeline-config` GET/PUT; `/api/llm-config` GET/PUT (AI only); `/api/properties/{id}/google/links/import` POST (GSC Links CSV); `PipelineRunnerFab` saves pipeline + LLM state before each run
+- **`web/`:** `/api/report/*` (PostgreSQL); `/api/run` spawns Python (localhost only); `/api/crawl/browser-status` GET (localhost, Playwright/Chromium preflight); `/api/pipeline-config` GET/PUT; `/api/llm-config` GET/PUT (AI only); `/api/chat` POST (SSE agent); `/api/chat/sessions` GET/POST; `/api/properties/{id}/google/links/import` POST (GSC Links CSV); `PipelineRunnerFab` saves pipeline + LLM state before each run
+- **MCP:** `python -m website_profiling.mcp` (stdio, **121 read-only audit tools** + MCP resources). See `docs/MCP.md`. Requires `pip install -r requirements-mcp.txt`.
+- **AI Chat UI:** `/chat` — property-scoped chat with saved sessions (`chat_sessions`, `chat_messages` tables, migration `012_chat_sessions`).
 - **Job store:** in-memory on `globalThis` in `web/src/server/pipelineJobs.ts` — job status/log is lost on server restart (single-process dev/Docker only).
 - **Docker:** `Dockerfile` + `docker-compose.yml` (postgres + web); **`docker-compose.pull.yml`** for pre-built images (`WEB_IMAGE`); **`LIGHTHOUSE_CHROME_FLAGS`**
 
@@ -36,7 +38,8 @@
 | Report | `reporting/builder.py`, `reporting/categories.py` |
 | DB schema | `alembic/versions/` |
 | Local analysis | `analysis/local.py`, `requirements.txt` |
-| AI insights (LLM) | `llm/enrich.py`, `llm_config.py`, `requirements-llm.txt` |
+| AI insights (LLM) | `llm/enrich.py`, `llm/agent.py`, `llm_config.py`, `requirements-llm.txt` |
+| Audit query tools (MCP + chat) | `tools/audit_tools/`, `mcp/server.py`, `commands/chat_cmd.py` |
 | Config / CLI | `config.py` (`load_config`, `load_config_from_db`), `cli.py`, `input.txt.example` |
 | UI pipeline schema | `web/src/lib/pipelineConfigSchema.ts` |
 | UI LLM schema | `web/src/lib/llmConfigSchema.ts` |

README.md

@@ -68,6 +68,15 @@ Google Search Console / Analytics: connect via **Integrations** (gear icon) in t
 
 **JavaScript crawl (optional):** In Audit settings, set **Crawl rendering** to `javascript` (always headless Chromium) or `auto` (static first, browser when SPA heuristics match). Install locally: `pip install -r requirements-browser.txt` and Chromium on `PATH` or `CHROME_PATH` (included in Docker). The UI preflights via `GET /api/crawl/browser-status` before runs when JS/auto is selected.
 
+**AI Chat (optional):** Ask questions about your audit data at [http://localhost:3000/chat](http://localhost:3000/chat). Enable a provider under **Run audit → AI settings** (`llm_enabled`, provider, model). `./local-run setup` installs `requirements-llm.txt` (`httpx`, OpenAI, Anthropic SDKs).
+
+| Provider | Notes |
+|----------|--------|
+| **Ollama** | Local daemon at `http://127.0.0.1:11434`. Chat UI lists installed models plus the live Ollama cloud catalog (billing: free local, account free tier, Pro). Native tool calling when supported; otherwise ReAct fallback. Pick the model in-chat without leaving the page. |
+| **OpenAI** / **Anthropic** | API key in AI settings; native tool calling with streaming. |
+
+The agent uses the same **121 read-only audit tools** as the MCP server (`docs/MCP.md`). Responses stream over SSE (`POST /api/chat`) with status, tool activity, and tokens. Sessions are saved per property (`chat_sessions` / `chat_messages`).
+
 Production: `docker-compose.prod.yml` (set `POSTGRES_PASSWORD`, `AUTH_SECRET`).
 
 ## License

alembic/versions/012_chat_sessions.py

@@ -0,0 +1,50 @@
+"""Chat sessions and messages for in-app AI chat.
+
+Revision ID: 012_chat_sessions
+Revises: 011_roadmap_foundation
+"""
+from __future__ import annotations
+
+from alembic import op
+
+revision = "012_chat_sessions"
+down_revision = "011_roadmap_foundation"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    op.execute("""
+        CREATE TABLE chat_sessions (
+            id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
+            property_id BIGINT NOT NULL REFERENCES properties(id) ON DELETE CASCADE,
+            title TEXT NOT NULL DEFAULT 'New chat',
+            created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+            updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
+        )
+    """)
+    op.execute("""
+        CREATE TABLE chat_messages (
+            id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
+            session_id BIGINT NOT NULL REFERENCES chat_sessions(id) ON DELETE CASCADE,
+            role TEXT NOT NULL CHECK (role IN ('user', 'assistant', 'tool')),
+            content TEXT NOT NULL DEFAULT '',
+            tool_name TEXT,
+            tool_args JSONB,
+            tool_result JSONB,
+            created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+        )
+    """)
+    op.execute("""
+        CREATE INDEX chat_sessions_property_updated_idx
+        ON chat_sessions(property_id, updated_at DESC)
+    """)
+    op.execute("""
+        CREATE INDEX chat_messages_session_created_idx
+        ON chat_messages(session_id, created_at)
+    """)
+
+
+def downgrade() -> None:
+    op.execute("DROP TABLE IF EXISTS chat_messages")
+    op.execute("DROP TABLE IF EXISTS chat_sessions")

docs/GLOSSARY.md

@@ -40,6 +40,8 @@ UI terms agencies recognize, mapped to internal keys and data sources.
 | Property alerts | `alert_webhook_url`, `/api/alerts/check` | Health snapshot rules | Ops notifications |
 | Content brief | Keywords Brief button, `/api/keywords/content-brief` | LLM or deterministic | Content planning |
 | AI issue fix | `llm_recommendation`, `/api/issues/fix-suggestion` | LLM on demand + report build | Actionable remediation |
+| AI Chat | `/chat`, `/api/chat`, `chat_sessions` | LLM + read-only audit tools | Conversational site audit queries |
+| MCP tools | `python -m website_profiling.mcp` | Same `audit_tools` as chat | Cursor / Claude Desktop integration — see [MCP.md](MCP.md) |
 | Read-only session | `AUTH_DEFAULT_ROLE=client-readonly`, `/api/auth/session` | Session cookie | Client view-only access |
 | Export executive summary | `export_audit_html/pdf/csv`, `executive_summary` | Report payload + optional AI | Client deliverable |
 

docs/MCP.md

@@ -0,0 +1,120 @@
+# Site Audit MCP server
+
+Read-only [Model Context Protocol](https://modelcontextprotocol.io) tools for querying audit data from Cursor, Claude Desktop, or other MCP clients.
+
+## Install
+
+```bash
+pip install -r requirements-mcp.txt
+export DATABASE_URL=postgres://profiling:profiling@localhost:5432/website_profiling
+export PYTHONPATH=src
+```
+
+## Cursor configuration
+
+Add to `.cursor/mcp.json` (or Cursor MCP settings):
+
+```json
+{
+  "mcpServers": {
+    "site-audit": {
+      "command": "python",
+      "args": ["-m", "website_profiling.mcp"],
+      "env": {
+        "DATABASE_URL": "postgres://profiling:profiling@localhost:5432/website_profiling",
+        "PYTHONPATH": "src",
+        "WP_PROPERTY_ID": "1"
+      }
+    }
+  }
+}
+```
+
+`WP_PROPERTY_ID` sets the default property when tools omit `property_id`.
+
+## MCP resources
+
+| URI | Content |
+|-----|---------|
+| `audit://properties` | JSON list of properties |
+| `audit://property/{id}` | Property details + latest report summary |
+| `audit://property/{id}/report/latest` | Payload key index (counts, not full blob) |
+| `audit://property/{id}/report/{report_id}` | Payload key index for a specific report |
+| `audit://glossary` | Excerpt from `docs/GLOSSARY.md` |
+| `audit://tools` | Tool catalog grouped by SEO domain |
+
+## Tools (121 read-only)
+
+### Portfolio and report
+
+`list_properties`, `get_property`, `get_report_summary`, `get_category_scores`, `get_executive_summary`, `get_report_meta`, `get_site_level`, `list_report_history`, `get_audit_recommendations`, `get_ml_errors`, `get_ssl_expiry_info`, `list_audit_categories`, `get_category_recommendations`, `get_crawl_summary`
+
+### Issues and workflow
+
+`list_issues`, `list_issues_by_category`, `get_category_issues`, `list_issue_workflow`, `list_issues_with_ai_fixes`, `list_seo_onpage_issues`
+
+### On-page SEO
+
+`list_content_url_issues`, `list_pages_missing_title`, `list_pages_missing_h1`, `list_pages_multiple_h1`, `list_pages_missing_meta_description`, `list_pages_meta_desc_too_short`, `list_pages_meta_desc_too_long`, `list_pages_noindex`, `get_seo_health`
+
+### Crawl and pages
+
+`search_pages`, `search_pages_advanced`, `get_page_details`, `get_page_analysis`, `get_internal_links`, `list_redirects`, `list_broken_links`, `list_status_4xx_pages`, `list_status_5xx_pages`, `get_status_code_breakdown`, `get_response_time_stats`, `get_depth_distribution`, `get_crawl_segments`, `get_browser_diagnostics_summary`, `list_pages_with_console_errors`, `list_pages_by_fetch_method`, `get_crawl_links_table`, `get_graph_edges_sample`
+
+### Schema and technical
+
+`get_schema_coverage`, `list_pages_without_schema`, `search_pages_by_schema_type`, `get_tech_stack_summary`, `get_security_findings`
+
+### Links and architecture
+
+`list_orphan_pages`, `get_top_linked_pages`, `get_top_crawled_pages`, `get_outbound_link_domains`, `get_link_graph_summary`, `get_url_fingerprints`, `get_mime_type_breakdown`, `get_title_length_distribution`, `get_domain_link_distribution`, `get_outlink_distribution`
+
+### Indexation and international
+
+`get_indexation_coverage`, `list_indexation_gaps`, `get_indexation_url_join`, `get_hreflang_summary`, `get_language_summary`
+
+### Content and social
+
+`get_content_analytics`, `get_content_duplicates`, `get_social_coverage`, `get_keyword_opportunities`, `get_ner_site_summary`, `list_thin_content_pages`
+
+### Keywords
+
+`get_keyword_summary`, `search_keywords`, `get_striking_distance_keywords`, `get_keyword_cannibalisation`, `get_query_page_misalignment`, `get_semantic_keyword_clusters`, `get_keyword_history`, `get_keyword_serp_overlay`, `list_keywords_by_action`, `list_keywords_by_position`, `list_keywords_by_impressions`
+
+### Google
+
+`get_google_summary`, `get_google_integration_status`, `get_gsc_top_queries`, `get_gsc_top_pages`, `get_ga4_summary`, `get_ga4_page_metrics`, `get_gsc_page_query_slice`
+
+### Backlinks
+
+`get_gsc_links_summary`, `get_gsc_links_import_status`, `get_gsc_sample_links`, `get_gsc_latest_links`, `get_third_party_links_overlay`, `get_backlinks_velocity`, `get_competitor_link_gap`, `get_bing_backlinks_summary`
+
+### Performance
+
+`get_lighthouse_summary`, `get_lighthouse_for_url`, `get_lighthouse_human_summary`, `get_lighthouse_diagnostics`, `get_crux_summary`, `list_slow_pages`, `list_lighthouse_poor_seo_pages`
+
+### Drift, health, and compare
+
+`get_health_history`, `get_category_health_history`, `compare_reports`, `compare_issue_deltas`, `compare_category_deltas`, `compare_seo_health_deltas`, `compare_lighthouse_deltas`, `compare_url_set_diff`, `compare_redirect_deltas`, `compare_link_metric_deltas`
+
+### Ops and logs
+
+`get_integration_alerts`, `get_property_ops`, `list_crawl_runs`, `list_log_uploads`, `get_latest_log_analysis`
+
+## Example prompts
+
+- "What indexation gaps exist between crawl and GSC?"
+- "List pages missing titles or meta descriptions"
+- "Show backlinks velocity and third-party overlay data"
+- "List keyword cannibalisation issues"
+- "Compare the latest audit to report ID 38 — what URLs were added or removed?"
+- "Which pages have JS console errors or poor Lighthouse SEO scores?"
+- "What does the latest access log analysis show?"
+
+## In-app chat
+
+The same tools power **AI Chat** at [http://localhost:3000/chat](http://localhost:3000/chat). Enable AI in Run audit → AI settings.
+
+## Ollama note
+
+When the local Ollama daemon supports native tools (most current models, including Ollama cloud refs like `minimax-m3:cloud`), chat uses Ollama’s `/api/chat` tool format. Older or tool-less models fall back to JSON ReAct parsing. OpenAI and Anthropic always use native tool calling with streaming in the chat UI.

local-run.ps1

@@ -0,0 +1 @@
+& "$PSScriptRoot\scripts\local-run.ps1" @args

local-test.ps1

@@ -0,0 +1,2 @@
+# Wrapper — run from repo root: .\local-test.ps1 [command]
+& "$PSScriptRoot\scripts\local-test.ps1" @args

requirements-mcp.txt

@@ -0,0 +1,2 @@
+# MCP server for Cursor / Claude Desktop (optional)
+mcp>=1.0.0