Mcp and chat

.github/workflows/ci.yml

@@ -46,7 +46,11 @@ jobs:
         run: |
           pytest tests/test_alert_checker.py tests/test_schedule_runner.py tests/test_export_audit.py \
             tests/test_export_audit_coverage.py tests/test_audit_tools.py tests/test_audit_tools_expanded.py \
-            tests/test_audit_tools_coverage.py \
+            tests/test_audit_tools_coverage.py tests/test_audit_tools_dispatch_coverage.py \
+            tests/test_export_custom_coverage.py tests/test_export_artifacts_coverage.py \
+            tests/test_export_compare_coverage.py tests/test_export_tools_coverage.py \
+            tests/test_image_tools.py tests/test_export_custom.py tests/test_export_artifacts.py \
+            tests/test_export_compare.py \
             tests/test_mcp_registry.py tests/test_mcp_resources.py \
             --cov=website_profiling.tools --cov-config=.coveragerc.tools \
             --cov-report=term-missing --cov-fail-under=95 -q -o addopts=

.gitignore

@@ -28,3 +28,4 @@ input.txt
 # Runtime shadow written to DATA_DIR by the web UI (not the committed example)
 pipeline-config.txt
 *__pycache__*
+.coverage

AGENT.md

@@ -12,7 +12,7 @@
 - `web/app/` -- routes; `web/src/` -- React; pipeline: `PipelineRunnerFab`, `server/pipelineJobs.ts`, `server/pipelineConfig.ts`, `server/llmConfig.ts`, `server/db.ts`
 - `alembic/` -- schema migrations
 
-**Local dev:** `./local-run` (Postgres in Docker `wp-pg`, Next.js on host). See `scripts/local-run.sh`. **Local tests (CI parity):** `./local-test` (100% in-scope coverage gate); `./local-test browser` for `@pytest.mark.browser` integration tests — see `scripts/local-test.sh`. Mocked browser unit tests: `tests/test_browser_fetcher_unit.py`.
+**Local dev:** `./local-run` (Postgres in Docker `wp-pg`, Next.js on host). See `scripts/local-run.sh`. **Local tests (CI parity):** `./local-test` runs **three** Python coverage gates (core 100%, reporting 100%, tools 95%); `./local-test browser` for `@pytest.mark.browser` integration tests — see `scripts/local-test.sh`. Mocked browser unit tests: `tests/test_browser_fetcher_unit.py`.
 
 **JavaScript crawl (optional):** Config keys `crawl_render_mode` (`static` | `javascript` | `auto`) and `crawl_js_*` in pipeline config / `pipelineConfigSchema.ts`. JS/auto crawls can capture browser console errors and uncaught exceptions (`crawl_js_capture_console`, stored under `page_analysis.browser`). **Auto mode** uses static-first fetch, pre-parse SPA heuristics (`needs_js_render`), then post-parse low-outlink fallback (`needs_js_render_after_parse`) in `crawler.py`. **Preflight:** `GET /api/crawl/browser-status` (localhost) spawns Python `browser_status()`; Run audit settings/run validation calls it when render mode is `javascript` or `auto`. Browser deps: `requirements-browser.txt` (installed by `./local-run setup` and `./local-test`). Runtime needs Chromium on `PATH` or `CHROME_PATH` (Docker sets `CHROME_PATH=/usr/bin/chromium`). Integration tests: `@pytest.mark.browser` — excluded by default in `pytest.ini`; Docker CI runs `tests/test_crawl_fetchers.py` and `tests/test_crawler_browser_e2e.py -m browser`; locally `./local-test browser`.
 
@@ -84,4 +84,35 @@ These recur when adding features. Verify explicitly — do not assume tests caug
    report_id = int(rid) if rid is not None else None
    ```
 
-**Checklist:** new report page uses `ReportShell` · no duplicate local imports in long functions · new `fetchone()` uses `_row_field`
+4. **Python — local vs CI coverage gates (three jobs, not one)**
+   - CI runs **three separate** pytest coverage jobs (see `.github/workflows/ci.yml` and `scripts/local-test.sh`):
+     | Gate | Config | Source | Threshold | Test scope |
+     |------|--------|--------|-----------|------------|
+     | Core | `.coveragerc` | all packages **except** `tools/` and `reporting/` | 100% | `pytest tests/ -m "not browser"` |
+     | Reporting | `.coveragerc.reporting` | `website_profiling.reporting` | 100% | fixed test file list |
+     | Tools | `.coveragerc.tools` | `website_profiling.tools` | 95% | fixed test file list |
+   - **Symptom:** `./local-test` or core pytest passes at 100%, but CI fails on tools/reporting (e.g. 84% tools).
+   - **Causes:** (a) only ran core pytest, not reporting/tools gates; (b) added tests under `tests/test_<module>_coverage.py` but did not add the file to the tools gate list in **both** `scripts/local-test.sh`, `scripts/local-test.ps1`, and `.github/workflows/ci.yml`; (c) changed code under `website_profiling/tools/` without tests that hit those lines in the tools gate subset.
+   - **Do:** Run full `./local-test` before push. When adding tools coverage tests, name them `tests/test_<module>_coverage.py` (repo convention) and register the file in all three places above. Keep bash and PowerShell local-test scripts in sync.
+   - **Don't:** Assume `pytest tests/` alone matches CI. Don't rely on a single mega `test_tools_coverage_gaps.py` — split by module.
+
+5. **Python — `runpy.run_module` / `__main__` guard tests**
+   - Tests that execute a module as `__main__` via `runpy.run_module(..., run_name="__main__")` emit:
+     `RuntimeWarning: '<module>' found in sys.modules after import of package ...`
+     when the same module was already imported at the top of the test file (or by another import).
+   - **Do:** Before `runpy.run_module`, remove the target from `sys.modules` so Python re-executes `__main__` cleanly. Name tests `test_module_main_guard` (see `tests/test_schedule_runner.py`).
+   - **Don't:** Call `runpy.run_module` on a module already imported in that test file without popping it first.
+
+   ```python
+   import runpy
+   import sys
+
+   sys.modules.pop("website_profiling.tools.schedule_runner", None)
+   runpy.run_module(
+       "website_profiling.tools.schedule_runner",
+       run_name="__main__",
+       alter_sys=False,
+   )
+   ```
+
+**Checklist:** new report page uses `ReportShell` · no duplicate local imports in long functions · new `fetchone()` uses `_row_field` · `./local-test` passes all three coverage gates · new tools coverage test file listed in CI + both local-test scripts · `runpy` main-guard tests pop `sys.modules` first

README.md

@@ -18,18 +18,6 @@ docker compose up --build
 
 Open [http://localhost:3000/home](http://localhost:3000/home).
 
-**Docker (published image)**
-
-The app requires PostgreSQL on the same Docker network. Do **not** run the image alone with `docker run` — the hostname `postgres` only resolves inside Compose.
-
-```bash
-docker pull your-registry/website-profiling:tag
-export WEB_IMAGE=your-registry/website-profiling:tag
-docker compose -f docker-compose.pull.yml up -d
-```
-
-Open [http://localhost:3000/home](http://localhost:3000/home).
-
 **Local dev**
 
 ```bash

docs/GLOSSARY.md

@@ -19,7 +19,7 @@ UI terms agencies recognize, mapped to internal keys and data sources.
 | Crawl summary | `charts` | Crawl aggregates | SF overview |
 | Internal links | `network` | Link graph | Ahrefs Internal Links |
 | Backlinks | `backlinks`, `gsc_links`, `gsc_links_data` | GSC Links CSV import (Google sample) | GSC Links report |
-| Page previews | `gallery` | Crawl excerpts | Visual QA |
+| Page previews | `gallery`, `list_site_image_urls`, `image_inventory` | Crawl excerpts + optional HTTP probe | Visual QA; size/format when probed |
 | Search Console | `search-performance`, `google_data` (scoped by `property_id`) | GSC API per property | Google Search Console |
 | Analytics (GA4) | `traffic`, `google_data` (scoped by `property_id`) | GA4 API per property | Google Analytics |
 | Keywords | `keywords-explorer`, `keyword_data` | Crawl + Search Console + research | Keyword tools (site-scoped) |
@@ -43,7 +43,7 @@ UI terms agencies recognize, mapped to internal keys and data sources.
 | AI Chat | `/chat`, `/api/chat`, `chat_sessions` | LLM + read-only audit tools | Conversational site audit queries |
 | MCP tools | `python -m website_profiling.mcp` | Same `audit_tools` as chat | Cursor / Claude Desktop integration — see [MCP.md](MCP.md) |
 | Read-only session | `AUTH_DEFAULT_ROLE=client-readonly`, `/api/auth/session` | Session cookie | Client view-only access |
-| Export executive summary | `export_audit_html/pdf/csv`, `executive_summary` | Report payload + optional AI | Client deliverable |
+| Export executive summary | `export_audit_html/pdf/csv`, `export_audit_report` (chat/MCP), Export view | Report payload + optional AI | Client deliverable |
 
 ## Metric names
 

docs/MCP.md

@@ -43,43 +43,55 @@ Add to `.cursor/mcp.json` (or Cursor MCP settings):
 | `audit://glossary` | Excerpt from `docs/GLOSSARY.md` |
 | `audit://tools` | Tool catalog grouped by SEO domain |
 
-## Tools (121 read-only)
+## Tools (171 read-only + export)
+
+### Export and deliverables
+
+`export_audit_report`, `export_compare_csv`, `export_list_as_csv`, `compose_custom_report`, `export_custom_report`, `list_export_formats`
+
+Full audit exports reuse the same generators as the Export view (PDF requires `reportlab`). Export tools store files as artifacts (24h TTL); in-app chat renders download buttons via `/api/chat/artifacts/{id}`.
+
+### Image audit
+
+`get_image_audit_summary`, `list_pages_without_lazy_images`, `list_pages_with_images_missing_dimensions`, `list_site_image_urls`, `list_lighthouse_image_opportunities`, `list_largest_images`, `list_unoptimized_images`, `list_images_needing_attention`
+
+Size-based tools require `probe_image_inventory=true` in pipeline config when building the report. Keys: `max_image_probe_urls` (default 500), `image_probe_concurrency`, `image_probe_timeout`, `image_unoptimized_min_kb` (default 200).
 
 ### Portfolio and report
 
-`list_properties`, `get_property`, `get_report_summary`, `get_category_scores`, `get_executive_summary`, `get_report_meta`, `get_site_level`, `list_report_history`, `get_audit_recommendations`, `get_ml_errors`, `get_ssl_expiry_info`, `list_audit_categories`, `get_category_recommendations`, `get_crawl_summary`
+`list_properties`, `get_property`, `get_report_summary`, `get_category_scores`, `get_executive_summary`, `get_report_meta`, `get_site_level`, `list_report_history`, `get_audit_recommendations`, `get_ml_errors`, `get_ssl_expiry_info`, `list_audit_categories`, `get_category_recommendations`, `get_crawl_summary`, `get_portfolio_summary`
 
 ### Issues and workflow
 
-`list_issues`, `list_issues_by_category`, `get_category_issues`, `list_issue_workflow`, `list_issues_with_ai_fixes`, `list_seo_onpage_issues`
+`list_issues`, `search_issues`, `list_issues_by_category`, `get_category_issues`, `list_issue_workflow`, `list_issues_with_ai_fixes`, `list_seo_onpage_issues`
 
 ### On-page SEO
 
-`list_content_url_issues`, `list_pages_missing_title`, `list_pages_missing_h1`, `list_pages_multiple_h1`, `list_pages_missing_meta_description`, `list_pages_meta_desc_too_short`, `list_pages_meta_desc_too_long`, `list_pages_noindex`, `get_seo_health`
+`list_content_url_issues`, `list_pages_missing_title`, `list_pages_missing_h1`, `list_pages_multiple_h1`, `list_pages_missing_meta_description`, `list_pages_meta_desc_too_short`, `list_pages_meta_desc_too_long`, `list_pages_noindex`, `get_seo_health`, `list_pages_missing_canonical`, `list_canonical_mismatch`, `list_pages_with_missing_alt`, `list_pages_skipped_headings`, `list_pages_missing_viewport`, `list_pages_missing_og_image`
 
 ### Crawl and pages
 
-`search_pages`, `search_pages_advanced`, `get_page_details`, `get_page_analysis`, `get_internal_links`, `list_redirects`, `list_broken_links`, `list_status_4xx_pages`, `list_status_5xx_pages`, `get_status_code_breakdown`, `get_response_time_stats`, `get_depth_distribution`, `get_crawl_segments`, `get_browser_diagnostics_summary`, `list_pages_with_console_errors`, `list_pages_by_fetch_method`, `get_crawl_links_table`, `get_graph_edges_sample`
+`search_pages`, `search_pages_advanced`, `get_page_details`, `get_page_analysis`, `get_internal_links`, `list_redirects`, `list_broken_links`, `list_status_4xx_pages`, `list_status_5xx_pages`, `get_status_code_breakdown`, `get_response_time_stats`, `get_depth_distribution`, `get_crawl_segments`, `get_browser_diagnostics_summary`, `list_pages_with_console_errors`, `list_pages_by_fetch_method`, `get_crawl_links_table`, `get_graph_edges_sample`, `list_long_redirect_chains`, `list_robots_blocked_urls`, `get_top_pages_by_pagerank`
 
 ### Schema and technical
 
-`get_schema_coverage`, `list_pages_without_schema`, `search_pages_by_schema_type`, `get_tech_stack_summary`, `get_security_findings`
+`get_schema_coverage`, `list_pages_without_schema`, `search_pages_by_schema_type`, `get_tech_stack_summary`, `list_pages_by_technology`, `get_security_findings`, `get_security_findings_summary`, `list_security_findings_by_type`
 
 ### Links and architecture
 
-`list_orphan_pages`, `get_top_linked_pages`, `get_top_crawled_pages`, `get_outbound_link_domains`, `get_link_graph_summary`, `get_url_fingerprints`, `get_mime_type_breakdown`, `get_title_length_distribution`, `get_domain_link_distribution`, `get_outlink_distribution`
+`list_orphan_pages`, `get_top_linked_pages`, `get_top_crawled_pages`, `get_outbound_link_domains`, `get_link_graph_summary`, `get_url_fingerprints`, `list_broken_link_sources`, `get_mime_type_breakdown`, `get_title_length_distribution`, `get_domain_link_distribution`, `get_outlink_distribution`
 
 ### Indexation and international
 
 `get_indexation_coverage`, `list_indexation_gaps`, `get_indexation_url_join`, `get_hreflang_summary`, `get_language_summary`
 
 ### Content and social
 
-`get_content_analytics`, `get_content_duplicates`, `get_social_coverage`, `get_keyword_opportunities`, `get_ner_site_summary`, `list_thin_content_pages`
+`get_content_analytics`, `get_content_duplicates`, `get_duplicate_cluster`, `get_social_coverage`, `get_keyword_opportunities`, `get_ner_site_summary`, `list_thin_content_pages`
 
 ### Keywords
 
-`get_keyword_summary`, `search_keywords`, `get_striking_distance_keywords`, `get_keyword_cannibalisation`, `get_query_page_misalignment`, `get_semantic_keyword_clusters`, `get_keyword_history`, `get_keyword_serp_overlay`, `list_keywords_by_action`, `list_keywords_by_position`, `list_keywords_by_impressions`
+`get_keyword_summary`, `search_keywords`, `get_striking_distance_keywords`, `get_keyword_cannibalisation`, `get_query_page_misalignment`, `get_semantic_keyword_clusters`, `get_keyword_history`, `get_keyword_serp_overlay`, `list_keywords_by_action`, `list_keywords_by_position`, `list_keywords_by_impressions`, `expand_keywords`, `generate_content_brief`
 
 ### Google
 
@@ -91,25 +103,41 @@ Add to `.cursor/mcp.json` (or Cursor MCP settings):
 
 ### Performance
 
-`get_lighthouse_summary`, `get_lighthouse_for_url`, `get_lighthouse_human_summary`, `get_lighthouse_diagnostics`, `get_crux_summary`, `list_slow_pages`, `list_lighthouse_poor_seo_pages`
+`get_lighthouse_summary`, `get_lighthouse_for_url`, `get_lighthouse_human_summary`, `get_lighthouse_diagnostics`, `get_crux_summary`, `list_slow_pages`, `list_lighthouse_poor_seo_pages`, `list_lighthouse_poor_accessibility_pages`, `list_lighthouse_poor_best_practices_pages`, `list_lighthouse_cwv_failures`
 
 ### Drift, health, and compare
 
-`get_health_history`, `get_category_health_history`, `compare_reports`, `compare_issue_deltas`, `compare_category_deltas`, `compare_seo_health_deltas`, `compare_lighthouse_deltas`, `compare_url_set_diff`, `compare_redirect_deltas`, `compare_link_metric_deltas`
+`get_health_history`, `get_category_health_history`, `compare_reports`, `compare_issue_deltas`, `compare_category_deltas`, `compare_seo_health_deltas`, `compare_lighthouse_deltas`, `compare_url_set_diff`, `compare_redirect_deltas`, `compare_link_metric_deltas`, `compare_security_deltas`, `compare_duplicate_deltas`, `compare_tech_deltas`, `compare_content_metrics`, `compare_google_metrics`, `compare_priority_counts`, `compare_health_score_delta`
 
 ### Ops and logs
 
-`get_integration_alerts`, `get_property_ops`, `list_crawl_runs`, `list_log_uploads`, `get_latest_log_analysis`
+`get_integration_alerts`, `get_property_ops`, `list_crawl_runs`, `list_log_uploads`, `get_latest_log_analysis`, `get_log_top_paths`, `list_log_only_paths`, `list_crawl_only_paths`, `get_log_googlebot_stats`, `get_log_analysis_by_id`, `get_page_coach`
+
+## Future pipeline items (not yet exposed as tools)
+
+These require additional crawl or third-party integrations before dedicated tools are useful:
+
+- Google Rich Results / schema validation API
+- Full backlink index and anchor-text analytics
+- axe / color-contrast accessibility audits
+- SERP rank tracking beyond GSC position snapshots
 
 ## Example prompts
 
 - "What indexation gaps exist between crawl and GSC?"
-- "List pages missing titles or meta descriptions"
-- "Show backlinks velocity and third-party overlay data"
-- "List keyword cannibalisation issues"
-- "Compare the latest audit to report ID 38 — what URLs were added or removed?"
-- "Which pages have JS console errors or poor Lighthouse SEO scores?"
-- "What does the latest access log analysis show?"
+- "List pages missing canonical tags or with canonical mismatches"
+- "Which paths appear in access logs but were not crawled?"
+- "Compare GSC clicks vs the previous audit"
+- "List pages failing Core Web Vitals thresholds"
+- "Show security finding changes since report 38"
+- "Which pages link to broken URLs?"
+- "Generate a content brief for keyword X"
+- "Download the audit as PDF"
+- "Export broken links as CSV"
+- "Compare report 38 to the current audit and give me a CSV diff"
+- "Build a client report with executive summary, category scores, and top critical issues as PDF"
+- "Which images are largest and unoptimized?"
+- "List pages with images missing alt or lazy loading"
 
 ## In-app chat
 

input.txt.example

@@ -41,6 +41,11 @@ max_nodes_plot = 400
 run_security_scan = true
 security_scan_active = false
 security_max_urls_probe = 20
+probe_image_inventory = false
+max_image_probe_urls = 500
+image_probe_concurrency = 6
+image_probe_timeout = 8
+image_unoptimized_min_kb = 200
 
 # --- Lighthouse ---
 lighthouse_url = 

pipeline-config.example.txt

@@ -42,6 +42,11 @@ max_nodes_plot = 400
 run_security_scan = true
 security_scan_active = false
 security_max_urls_probe = 20
+probe_image_inventory = false
+max_image_probe_urls = 500
+image_probe_concurrency = 6
+image_probe_timeout = 8
+image_unoptimized_min_kb = 200
 
 # --- Lighthouse ---
 lighthouse_url = 

scripts/local-test.ps1

@@ -254,6 +254,15 @@ function Invoke-PytestTools {
         tests/test_audit_tools.py `
         tests/test_audit_tools_expanded.py `
         tests/test_audit_tools_coverage.py `
+        tests/test_audit_tools_dispatch_coverage.py `
+        tests/test_export_custom_coverage.py `
+        tests/test_export_artifacts_coverage.py `
+        tests/test_export_compare_coverage.py `
+        tests/test_export_tools_coverage.py `
+        tests/test_image_tools.py `
+        tests/test_export_custom.py `
+        tests/test_export_artifacts.py `
+        tests/test_export_compare.py `
         tests/test_mcp_registry.py `
         tests/test_mcp_resources.py `
         --cov=website_profiling.tools `