Skip to content

Commit d9b72bb

Browse files
codefrydevcodefrydev
authored
ok
2 parents a852742 + f7943d7 commit d9b72bb

373 files changed

Lines changed: 27122 additions & 9650 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.coverage‎

52 KB
Binary file not shown.

‎.coveragerc‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
[run]
2+
source = website_profiling
3+
4+
# We are NOT hiding core code. We only omit modules that require external
5+
# services/binaries (Google APIs, Lighthouse) or are impractical to unit-test here.
6+
omit =
7+
*/website_profiling/integrations/google/*
8+
*/website_profiling/lighthouse/*
9+
*/website_profiling/reporting/*
10+
*/website_profiling/tools/*
11+
*/website_profiling/security_scanner.py
12+
*/website_profiling/llm/providers/*
13+
*/website_profiling/llm/*
14+
*/website_profiling/llm_config.py
15+
*/website_profiling/cli.py
16+
*/website_profiling/commands/enrich_cmd.py
17+
18+
[report]
19+
show_missing = True
20+
skip_empty = True
21+

‎.github/ISSUE_TEMPLATE/bug_report.yml‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
name: Bug report
2+
description: Something is broken or incorrect
3+
title: "[Bug]: "
4+
labels: ["bug"]
5+
body:
6+
- type: markdown
7+
attributes:
8+
value: |
9+
Thanks for reporting. Please search existing issues first.
10+
Do **not** use this form for security vulnerabilities — see [SECURITY.md](../../SECURITY.md).
11+
- type: textarea
12+
id: description
13+
attributes:
14+
label: What happened?
15+
description: Clear description of the bug
16+
validations:
17+
required: true
18+
- type: textarea
19+
id: expected
20+
attributes:
21+
label: What did you expect?
22+
validations:
23+
required: true
24+
- type: textarea
25+
id: reproduce
26+
attributes:
27+
label: Steps to reproduce
28+
placeholder: |
29+
1. Run ./local-run
30+
2. Open Overview for example.com
31+
3. See error ...
32+
validations:
33+
required: true
34+
- type: dropdown
35+
id: area
36+
attributes:
37+
label: Area
38+
options:
39+
- Web UI
40+
- Python / crawl / report
41+
- Database / migrations
42+
- Google integrations
43+
- Docker / deploy
44+
- Other
45+
validations:
46+
required: true
47+
- type: input
48+
id: version
49+
attributes:
50+
label: Commit or version (if known)
51+
placeholder: git SHA or tag
52+
- type: textarea
53+
id: logs
54+
attributes:
55+
label: Logs or screenshots
56+
description: Paste relevant errors or attach screenshots

‎.github/ISSUE_TEMPLATE/config.yml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
blank_issues_enabled: false
2+
contact_links:
3+
- name: Contributing guide
4+
url: https://github.com/codefrydev/WebsiteProfiling/blob/master/CONTRIBUTING.md
5+
about: Setup, tests, and pull request expectations
6+
- name: Code of Conduct
7+
url: https://github.com/codefrydev/WebsiteProfiling/blob/master/CODE_OF_CONDUCT.md
8+
about: Community standards

‎.github/ISSUE_TEMPLATE/feature_request.yml‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
name: Feature request
2+
description: Suggest an improvement or new capability
3+
title: "[Feature]: "
4+
labels: ["enhancement"]
5+
body:
6+
- type: markdown
7+
attributes:
8+
value: |
9+
Describe the problem you are trying to solve. Feature requests that fit the project goal (free, honest technical SEO audits) are most likely to be accepted.
10+
- type: textarea
11+
id: problem
12+
attributes:
13+
label: Problem
14+
description: What limitation or pain point does this address?
15+
validations:
16+
required: true
17+
- type: textarea
18+
id: solution
19+
attributes:
20+
label: Proposed solution
21+
validations:
22+
required: true
23+
- type: textarea
24+
id: alternatives
25+
attributes:
26+
label: Alternatives considered
27+
- type: checkboxes
28+
id: terms
29+
attributes:
30+
label: Contribution
31+
options:
32+
- label: I am willing to help implement this via a pull request
33+
required: false

‎.github/pull_request_template.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
## Summary
2+
3+
<!-- What does this PR change and why? -->
4+
5+
## Type of change
6+
7+
- [ ] Bug fix
8+
- [ ] New feature
9+
- [ ] Documentation
10+
- [ ] Refactor / chore
11+
- [ ] Database migration
12+
13+
## How to test
14+
15+
<!-- Steps for reviewers, e.g. ./local-run, run audit on X, expect Y -->
16+
17+
## Checklist
18+
19+
- [ ] I read [CONTRIBUTING.md](../CONTRIBUTING.md) and followed it
20+
- [ ] `./local-test` or an equivalent subset passes locally
21+
- [ ] UI copy updates are in `web/src/strings.json` (if applicable)
22+
- [ ] No secrets or credentials in the diff
23+
- [ ] Alembic migration included if the schema changed
24+
25+
## Screenshots (UI only)
26+
27+
<!-- If applicable -->

‎.github/workflows/ci.yml‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches: [ master]
6+
pull_request:
7+
8+
jobs:
9+
python:
10+
runs-on: ubuntu-latest
11+
services:
12+
postgres:
13+
image: postgres:16-alpine
14+
env:
15+
POSTGRES_USER: profiling
16+
POSTGRES_PASSWORD: profiling
17+
POSTGRES_DB: website_profiling
18+
ports:
19+
- 5432:5432
20+
options: >-
21+
--health-cmd "pg_isready -U profiling -d website_profiling"
22+
--health-interval 5s
23+
--health-timeout 3s
24+
--health-retries 5
25+
env:
26+
DATABASE_URL: postgres://profiling:profiling@localhost:5432/website_profiling
27+
steps:
28+
- uses: actions/checkout@v4
29+
- uses: actions/setup-python@v5
30+
with:
31+
python-version: '3.12'
32+
- name: Install dependencies
33+
run: pip install -r requirements.txt
34+
- name: Apply migrations
35+
run: alembic upgrade head
36+
- name: Pytest
37+
run: pytest tests/ -q
38+
- name: CLI smoke
39+
run: python -m src --help
40+
41+
docker:
42+
runs-on: ubuntu-latest
43+
steps:
44+
- uses: actions/checkout@v4
45+
- name: Build image
46+
run: docker build -t website-profiling:ci .
47+
48+
web:
49+
runs-on: ubuntu-latest
50+
defaults:
51+
run:
52+
working-directory: web
53+
steps:
54+
- uses: actions/checkout@v4
55+
- uses: actions/setup-node@v4
56+
with:
57+
node-version: '20'
58+
cache: npm
59+
cache-dependency-path: web/package-lock.json
60+
- name: Install
61+
run: npm ci
62+
- name: Typecheck
63+
run: npm run typecheck
64+
- name: Lint
65+
run: npm run lint
66+
- name: Test
67+
run: npm test

‎.gitignore‎

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,19 +2,29 @@
22

33
# Next.js UI: generated pipeline configs from the runner modal (repo root; must match Python cwd for paths)
44
.website-profiling-ui-*.txt
5-
# Legacy: configs under subfolder broke sqlite_db = report.db vs Next (kept for ignore if present)
5+
# Legacy UI pipeline configs under subfolder (kept for ignore if present)
66
.web-pipeline/
77

8-
# WebsiteProfiling generated outputs
8+
# Local runtime data (shadow config, secrets volume mount)
9+
data/*
10+
!data/.gitkeep
11+
12+
# Python virtual environments (README recommends .venv; ignore both names at repo root)
13+
.venv/
14+
venv/
15+
16+
# Legacy SQLite outputs (if present locally)
917
report.db
1018
report.db.*
11-
1219
report.db-journal
1320
nodes.json
1421

1522
# Google integration secrets (never commit)
1623
.secrets/
1724
web/.env.local
1825

19-
# Legacy local config file (use report.db / web UI instead)
26+
# Legacy local config file (use PostgreSQL pipeline_config / web UI instead)
2027
input.txt
28+
# Runtime shadow written to DATA_DIR by the web UI (not the committed example)
29+
pipeline-config.txt
30+
*__pycache__*

‎AGENT.md‎

Lines changed: 28 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,34 +1,45 @@
1-
# Agent instructions -- WebsiteProfiling
1+
# Agent instructions — Site Audit (WebsiteProfiling)
22

3-
**What it is:** `python -m src` from repo root (`src/__main__.py` -> package **`website_profiling`**). Config: stored in **`report.db`** (`pipeline_config` table, `key/value/is_unknown/updated_at`). A shadow **`pipeline-config.txt`** is auto-written next to `report.db` on every Save/Run. CLI loads DB first (`REPORT_DB_PATH` or `cwd/report.db`), then shadow file; `--config` overrides with a file. Reference keys: `input.txt.example` (not auto-loaded).
3+
**What it is:** `python -m src` from repo root (`src/__main__.py` -> package **`website_profiling`**). Config: stored in **PostgreSQL** (`pipeline_config` table, `key/value/is_unknown/updated_at`). A shadow **`pipeline-config.txt`** is auto-written to `DATA_DIR` on every Save/Run. CLI loads DB first (`DATABASE_URL`), then shadow file; `--config` overrides with a file. Reference keys: `input.txt.example` and `pipeline-config.example.txt` (not auto-loaded).
44

5-
**Frontend:** **`web/`** (Next.js) -- server reads `report.db` via `/api/report/*`.
5+
**LLM / AI:** Settings live in **`llm_config`** table in PostgreSQL. Configure only via web UI **AI** tab (`GET/PUT /api/llm-config`, localhost). Never in `pipeline-config.txt` or `--config`.
6+
7+
**Frontend:** **`web/`** (Next.js) -- server reads PostgreSQL via `/api/report/*`.
68

79
**Key paths**
810

9-
- `src/website_profiling/` -- `cli.py`, `config.py`, `crawl/`, `db/storage.py`, `lighthouse/`, `reporting/`, `ml/enrich.py`, `tools/`
10-
- `web/app/` -- routes; `web/src/` -- React; pipeline: `PipelineRunnerFab`, `server/pipelineJobs.js`, `server/pipelineConfig.js`
11+
- `src/website_profiling/` -- `cli.py`, `config.py`, `crawl/`, `db/storage.py`, `lighthouse/`, `reporting/`, `analysis/`, `llm/`, `tools/`
12+
- `web/app/` -- routes; `web/src/` -- React; pipeline: `PipelineRunnerFab`, `server/pipelineJobs.ts`, `server/pipelineConfig.ts`, `server/llmConfig.ts`, `server/db.ts`
13+
- `alembic/` -- schema migrations
14+
15+
**Local dev:** `./local-run` (Postgres in Docker `wp-pg`, Next.js on host). See `scripts/local-run.sh`. **Local tests (CI parity):** `./local-test` — see `scripts/local-test.sh`.
1116

1217
**Run / APIs**
1318

14-
- Pipeline: `python -m src` — reads config from `report.db` (`pipeline_config`); shadow `pipeline-config.txt` if table empty. CLI override: `python -m src --config path`
15-
- Optional step: `crawl` | `report` | `plot` | `lighthouse` | `keywords` | `warnings` | `enrich`
16-
- **`preserve_crawl_history`** (default true): append crawls; `false` recreates crawl tables but restores `report_payload`, Lighthouse, `google_data`, `keyword_data`, `keyword_history`, `keyword_suggest_cache`, and `crawl_runs`
17-
- **`enrich_keywords_after_report`**: when omitted or `auto` (UI: Auto), follows `enable_google_search_console`; when set to Yes/No, explicit override
18-
- **`REPORT_DB_PATH`** env: DB path used by both Python and Next.js (Docker: `/data/report.db`; local default: `report.db` at repo root). Pipeline config lives in this DB.
19-
- **`web/`:** `/api/report/*` (SQLite); `/api/run` spawns Python (localhost only); `/api/pipeline-config` GET/PUT for persistent settings; `PipelineRunnerFab` saves state to `report.db` (`pipeline_config` table) + shadow `pipeline-config.txt` before each run
20-
- **Docker:** `Dockerfile` + `docker-compose.yml`; **`LIGHTHOUSE_CHROME_FLAGS`**; ML caches under `/data/cache/*` in compose
19+
- Run audit (CLI): `python -m src` — reads config from PostgreSQL (`pipeline_config`); shadow `DATA_DIR/pipeline-config.txt` if table empty. CLI override: `python -m src --config path`
20+
- Optional step: `crawl` | `report` | `plot` | `lighthouse` | `keywords` | `warnings` | `enrich` | `google`
21+
- **`preserve_crawl_history`** (default true): append crawls; `false` truncates crawl tables but restores `report_payload`, Lighthouse, `google_data`, `keyword_data`, `keyword_history`, `keyword_suggest_cache`, and `crawl_runs`
22+
- **`DATABASE_URL`** env: PostgreSQL connection string (required). **`DATA_DIR`**: secrets + shadow config (Docker: `/data`).
23+
- **Pipeline data** (crawl, edges, nodes, report payload, Lighthouse, keywords, warnings) is stored in **PostgreSQL only** — no JSON/CSV/HTML exports from the main pipeline.
24+
- **Pool tuning:** `DB_POOL_MIN` / `DB_POOL_MAX` (Python), `PGPOOL_MAX` (Node). Bulk crawl writes via `executemany`; optional **`crawl_stream_to_db`** streams rows during fetch.
25+
- **`web/`:** `/api/report/*` (PostgreSQL); `/api/run` spawns Python (localhost only); `/api/pipeline-config` GET/PUT; `/api/llm-config` GET/PUT (AI only); `PipelineRunnerFab` saves pipeline + LLM state before each run
26+
- **Job store:** in-memory on `globalThis` in `web/src/server/pipelineJobs.ts` — job status/log is lost on server restart (single-process dev/Docker only).
27+
- **Docker:** `Dockerfile` + `docker-compose.yml` (postgres + web); **`LIGHTHOUSE_CHROME_FLAGS`**
2128

2229
**Where to edit**
2330

2431
| Task | Where |
2532
|------|--------|
2633
| Crawl | `crawl/crawler.py` |
2734
| Report | `reporting/builder.py`, `reporting/categories.py` |
28-
| DB schema | `db/storage.py` `init_schema` |
29-
| ML | `ml/enrich.py`, `requirements-ml.txt` |
35+
| DB schema | `alembic/versions/` |
36+
| Local analysis | `analysis/local.py`, `requirements.txt` |
37+
| AI insights (LLM) | `llm/enrich.py`, `llm_config.py`, `requirements-llm.txt` |
3038
| Config / CLI | `config.py` (`load_config`, `load_config_from_db`), `cli.py`, `input.txt.example` |
31-
| UI config schema | `web/src/lib/pipelineConfigSchema.js` |
32-
| UI config I/O | `web/src/server/pipelineConfig.js` |
39+
| UI pipeline schema | `web/src/lib/pipelineConfigSchema.ts` |
40+
| UI LLM schema | `web/src/lib/llmConfigSchema.ts` |
41+
| UI config I/O | `web/src/server/pipelineConfig.ts`, `web/src/server/llmConfig.ts` |
42+
43+
Schema changes: add Alembic migration (`alembic revision`).
3344

34-
Schema changes: edit `init_schema` only (no migration layer). ML stack: prefer Python **3.12** for spaCy/blis; **3.13** may fail pip builds.
45+
**Company standards:** UI copy in `web/src/strings.json` (Site Audit, Properties, Run audit). Data provenance on `report_meta` in report payload. Docs: `docs/COMPANY_STANDARDS.md`, `docs/GLOSSARY.md`. Migration `003_company_standards` (properties, pipeline_jobs, audit_log). Durable jobs in `web/src/server/pipelineJobsDb.ts`. Export: `GET /api/report/export`, `src/website_profiling/tools/export_audit.py`.

0 commit comments

Comments
 (0)