feat(pge): intrinsic-eval toolkit + ontology-gen Evaluator + in-app scorecard + Auto-Map run-visualizer

.gitignore

@@ -80,10 +80,11 @@ mlartifacts/
 # Cookie jars / curl session dumps (may contain live session + CSRF tokens)
 cookies*.txt
 *.cookiejar
-tests/e2e/_e2e_server.log
 
+# Local git worktrees for isolated feature work
+.worktrees/
+
+tests/e2e/_e2e_server.log
 
 .claude/
 .understand-anything/
-
-

README.md

@@ -256,6 +256,39 @@ require installing the optional extra:
 uv sync --extra pitfalls
 ```
 
+### PGE Intrinsic Evaluation (quality scorecard)
+
+The PGE pipeline (ontology + mapping generation) is scored by a
+**usecase-agnostic, gold-free** scorecard (`src/agents/pge_eval/`). It uses
+intrinsic structural/self-consistency metrics — table/column footprint
+coverage, orphan classes, dangling domain/range, naming/duplicate hygiene,
+mapping completeness, id-integrity, dangling-FK fractions — across three gate
+tiers (absolute / ratio / self-baseline regression), plus an **advisory**
+LLM-judge that never gates. No domain reference answer is encoded, so it works
+for any domain.
+
+It runs in two places:
+
+- **In-app** — after you generate an ontology or mappings, a scorecard is
+  attached to the task result (`pge_scorecard`) and the verdict is shown in the
+  completion message (e.g. `… · quality GREEN`). Deterministic, no extra LLM.
+- **CLI** — `scripts/goals_eval.py`:
+
+  ```bash
+  # Score a captured artifact (offline, deterministic; --no-judge = zero network)
+  .venv/bin/python scripts/goals_eval.py score <artifact.json> [--no-judge] [--gate-ratios]
+
+  # Run the pipeline live for ANY domain, then score it (domain-agnostic)
+  .venv/bin/python scripts/goals_eval.py run --registry-json <domain-export.json> [--version V]
+  .venv/bin/python scripts/goals_eval.py run --ontology <ontology.json> --metadata <metadata.json>
+  ```
+
+  Exit code is the verdict: `0` GREEN, non-zero RED.
+
+The OWL generator also gains an **Evaluator stage**: the Stage-1 deterministic
+checks run in its generation loop and feed retry-hints back (a real PGE loop),
+bounded by a retry cap.
+
 ### Documentation
 
 Full documentation is available in [`docs/`](docs/README.md). For a comprehensive feature list and architecture details, see [INFO.md](docs/INFO.md).

changelogs/v0.5.0/2026-06-02.log

@@ -0,0 +1,108 @@
+# v0.5.0 — 2026-06-02
+
+## PGE generation quality: exhaustive attributes + value harmonization
+
+### Context
+`newdomain` (ontobricks_pge_registry) was attribute-sparse: the OWL generator
+emitted a curated ~28-property ontology subset (vs V1.1's ~53), so most classes
+had empty `dataProperties` and the EntityGenerator produced ID+Label-only
+entities — unfit as the KPI/analytical source of truth. Root cause is generation
+QUALITY in two LLM-driven agents, not plumbing (the sync/finalize infra in
+`Ontology.finalize_class_attributes` already works and runs at session load).
+This change strengthens both generators so the PGE pipeline reproduces V1.1-level
+coverage from source introspection.
+
+### Changes
+1. `src/agents/agent_owl_generator/engine.py` — Part 1: exhaustive datatype-property
+   coverage.
+   - Added a `# ATTRIBUTE COVERAGE (CRITICAL — exhaustive, NOT curated)` section
+     to the system prompt: emit a DatatypeProperty for EVERY meaningful source
+     column per class across ALL covering trust tables; exclude only surrogate
+     keys / audit columns / FK columns; collapse cross-trust synonyms to one
+     property; lowerCamelCase `[a-z][A-Za-z0-9]*` names; rich clinical entities
+     warrant 6–11 properties; "2 props" is a floor not a target.
+   - Workflow now instructs `get_table_detail` on every covering table (so the
+     LLM sees the full column list past the 80-column get_metadata truncation).
+   - Softened the conflicting `GENERIC_GUIDELINES §2.1` minimalism line to defer
+     to the new coverage rule.
+2. `src/agents/agent_mapping_pge/generators/entity.py` — Part 2: value harmonization
+   + regex safety.
+   - Added a `VALUE HARMONIZATION` section: for coded attributes (method/status/
+     type/mode/outcome), discover raw distinct values (`SELECT DISTINCT`) then map
+     to ONE canonical lowercase token set with a CASE expression aliased to the
+     clean attribute name, using the SAME tokens across all UNION branches
+     (delivery method → caesarean/instrumental/vaginal worked example).
+   - Added a `REGEX SAFETY` section: always `[0-9]`/`[a-z]`, NEVER `\d`/`\w`/`\s`
+     (the build strips a lone backslash → `\d` degrades to literal `d`).
+   - Workflow step 3 now references harmonization.
+3. `src/agents/agent_mapping_pge/planner.py` — Part 2: multi-trust completeness.
+   - Added a COMPLETENESS rule to CANONICAL-KEY NORMALIZATION: include EVERY
+     covering trust table in `canonical_column_per_table`, not just the two
+     checked for overlap (omitting one drops 30–60% of instances and dangles
+     relationships). Planner already emits `[0-9]`-safe regex and mandates
+     multi-trust UNION — this reinforces full coverage during candidate discovery.
+4. `tests/agents/agent_mapping_pge/test_entity_generator.py` — added
+   `test_system_prompt_mandates_value_harmonization` (canonical token set,
+   discover-before-harmonize, `[0-9]`-not-`\d` regex safety).
+
+### Modified files
+- src/agents/agent_owl_generator/engine.py
+- src/agents/agent_mapping_pge/generators/entity.py
+- src/agents/agent_mapping_pge/planner.py
+- tests/agents/agent_mapping_pge/test_entity_generator.py
+
+### Test result
+`uv run --offline pytest tests/agents/agent_mapping_pge/ tests/units/ontology/test_owl_generator.py`
+— 101 passed → 102 passed with the new harmonization test. No regressions.
+
+---
+
+# v0.5.0 — 2026-06-03
+
+## OWL generator output ceiling + live newdomain regeneration
+
+### Context
+Exercising the v0.5.0 generation-quality changes end-to-end against the live
+`ontobricks-pge` registry surfaced one defect and confirmed the rest.
+
+### Changes
+1. `src/agents/agent_owl_generator/engine.py` — raised the LLM completion ceiling
+   from a hardcoded `max_tokens=4096` to a named `MAX_OUTPUT_TOKENS = 16000`
+   (both the main-loop and tools-unsupported fallback call sites). The exhaustive
+   `# ATTRIBUTE COVERAGE` prompt makes the Turtle output larger; at 4096 the final
+   statement was silently truncated (`finish_reason=length`) and OWL parsing failed.
+   Claude Opus supports large completions; 16k fits a full maternity ontology.
+
+### Live regeneration result (newdomain, ontobricks-pge / ontobricks_pge_registry)
+Ran the real PGE pipeline against the live workspace (endpoint
+`databricks-claude-opus-4-7`, warehouse `e6b70b0c07bbaa10`,
+`fiifi_cdm_demo_catalog`):
+- OWL generator → 17 classes, **60 datatype properties** (was 22), all KPI props.
+- Mapping-PGE → 15 entities + 12 relationships, every item PASS the deterministic
+  evaluator, **72 attribute mappings**, multi-trust UNIONs + value harmonization,
+  `[0-9]`-safe canonical key, GROUP BY dedup. (ClinicalFinding / ClinicalProvider
+  honestly skipped — no source rows; they were ID+Label stubs in V1.1.)
+- One targeted re-run of Pregnancy via a retry-hint to DERIVE `bookingGestationWeeks`
+  (= `DATEDIFF(lmp_date, booking_date)/7`, a computed feature with no source column).
+- Overwrote `newdomain` v1 in Lakebase (rollback backup
+  `/tmp/newdomain_live_backup_2026-06-02.json`).
+
+### Independent verification (warehouse audit, round-tripped from Lakebase)
+- Entity-join dangling: **0.00% across all 12 relationships** (worst case 0.00%).
+- Delivery method mix vaginal 1227 / caesarean 849 / instrumental 297 (matches V1.1).
+- Pregnancy outcome delivered 1658 / transferred 715 / ongoing 74; feeding_status
+  mixed 549 / formula 543 / breast 542; booking gestation avg 10.5 wks; Pregnancy
+  hub 2463 rows = 2463 distinct IDs.
+- All 5 KPI columns present: deliveryMethod, outcomeStatus, bookingGestationWeeks,
+  contactType (=contact_type), feedingStatus (=feeding_status).
+
+NOTE (next phase, out of scope here): the downstream `canonical.*` UC views +
+metric views were built from the prior hand-copy's entity SQLs and must be
+republished from the regenerated newdomain SQLs.
+
+### Modified files
+- src/agents/agent_owl_generator/engine.py
+
+### Test result
+`uv run --offline pytest tests/agents/agent_mapping_pge/ tests/units/ontology/test_owl_generator.py`
+— 102 passed. No regressions.

changelogs/v0.5.0/FiifiB_2026-06-16.log

@@ -0,0 +1,73 @@
+# v0.5.0 — 2026-06-16 (FiifiB)
+
+## PGE intrinsic-evaluation toolkit + ontology-gen Evaluator stage + in-app scorecard
+
+### Context
+The PGE pipeline (ontology + mapping generation) had no measurable, gold-free
+stopping condition. This adds a **usecase-agnostic intrinsic scorecard**
+(`src/agents/pge_eval/`), wires it to run **in-app** after generation/mapping,
+turns `agent_owl_generator` into a real PGE loop via a new Evaluator stage, and
+ships a CLI. No golden/reference labels are encoded (correctness is measured by
+internal consistency against the actual runtime inputs), so it works for any
+domain — not just the NHS CDM/maternity demo it was developed against.
+
+### Changes
+1. `src/agents/pge_eval/` (new package) — the scorer:
+   - `scorecard.py` — `score_artifact`, the single offline-testable scoring
+     core; emits the scorecard JSON + GREEN/RED verdict + exit code.
+   - `ontology_metrics.py` / `mapping_metrics.py` / `pipeline_metrics.py` —
+     Stage-1 (footprint coverage, orphan/dangling/naming/duplicate), Stage-2
+     (entity/rel/attribute completeness, dangling-FK max, id-integrity, sql-exec
+     failures, conditional cross-source band), and pipeline (coverage_loss +
+     convergence, advisory/tracked only).
+   - `gates.py` — three tiers: absolute / ratio (`--gate-ratios`) / direction-
+     aware self-baseline regression.
+   - `baseline.py` — Tier-3 self-baseline store in `logs/goals/` (baseline = most
+     recent GREEN; RED runs never baseline).
+   - `judge.py` — advisory LLM-judge; the only network path; never gates; skipped
+     under `--no-judge` (zero network).
+   - `normalize.py` / `loaders.py` — shape-agnostic ontology/metadata
+     normalisation + domain-agnostic live-run input loaders.
+   - `inapp.py` — fail-safe in-app hooks (deterministic, never raise).
+2. `scripts/goals_eval.py` (new) — CLI with `score` / `run` subcommands and
+   `--no-judge` / `--gate-ratios`; `run` is domain-agnostic
+   (`--registry-json`/`--version` or `--ontology`/`--metadata`). Exit code =
+   verdict.
+3. `src/agents/agent_owl_generator/engine.py` — new **Evaluator stage**: after
+   the (upstream) pitfall-tool loop settles, run the Stage-1 deterministic checks
+   and feed retry-hints back into generation, bounded by `MAX_OWL_EVAL_ROUNDS`;
+   cleans prose/markdown preambles before parsing; de-maternified the ATTRIBUTE
+   COVERAGE prompt (domain-neutral examples).
+4. In-app scorecard hooks — `src/api/routers/internal/ontology.py` and
+   `src/back/objects/mapping/Mapping.py`: after generation/mapping, attach
+   `pge_scorecard` to the task result and append the verdict to the completion
+   message (kept alongside upstream's per-iteration generation score).
+5. Generality hardening — de-maternified the mapping VALUE HARMONIZATION
+   (`generators/entity.py`) and planner CANONICAL-KEY NORMALIZATION
+   (`planner.py`) prompts; broadened `test_no_domain_hardcoding` to also scan the
+   generator prompt files.
+6. `scripts/smoke_pge.py` — artifact dump embeds `ontology`/`metadata`/
+   `elapsed_s` so score-only is self-contained.
+7. Docs — Sphinx `docs/sphinx/api/agents.rst` "PGE Intrinsic Evaluation" section
+   (all 11 modules) + README "PGE Intrinsic Evaluation (quality scorecard)"
+   section.
+8. Integrated 65 upstream `master` commits (mid-0.5.0) via merge `198f38e`;
+   re-landed the Evaluator stage onto upstream's new tool-driven pitfall loop.
+
+### Modified / added files
+- src/agents/pge_eval/*.py (new: __init__, scorecard, normalize, ontology_metrics,
+  mapping_metrics, pipeline_metrics, gates, baseline, judge, inapp, loaders)
+- scripts/goals_eval.py (new), scripts/smoke_pge.py
+- src/agents/agent_owl_generator/engine.py
+- src/agents/agent_mapping_pge/generators/entity.py, src/agents/agent_mapping_pge/planner.py
+- src/api/routers/internal/ontology.py, src/back/objects/mapping/Mapping.py
+- docs/sphinx/api/agents.rst, README.md
+- tests/units/pge_eval/* (new), tests/agents/agent_mapping_pge/{test_entity_generator,test_planner}.py
+
+### Test result
+- `pytest tests/units tests/agents` → **2528 passed, 11 skipped, 0 failed**
+  (post-merge with origin/master).
+- Evidence captured during development: clean artifact → GREEN/exit 0; seeded
+  dangling-FK → RED/exit 1; Tier-3 regression → RED; `--no-judge` zero-network
+  verified; grep clean for trust_a/b/c, preg, maternity, NHS, spr in scorer +
+  generator prompts.

changelogs/v0.5.0/FiifiB_2026-06-19.log

@@ -0,0 +1,67 @@
+# v0.5.0 — PGE Run-Visualizer (make the mapping PGE loop demoable in the UI)
+
+## Context
+
+The mapping PGE pipeline (Planner → Generator → Evaluator → Critic) already
+computes rich artifacts — the planner's `source_model`, per-item
+`mapping_evaluations` (EvalReports), an attempt-by-attempt `mapping_run_log`,
+and an intrinsic-eval `pge_scorecard` — but the UI surfaced almost none of it.
+Auto-Map showed a progress bar, a flat per-item results table, and appended
+the scorecard verdict to a toast string. The orchestration (planner picks,
+generator attempts, evaluator/critic verdicts, retry hints, re-plan
+escalations) was invisible, which made the agentic loop "hard to demo on the
+OntoBricks UI."
+
+This change exposes the PGE loop in the Batch Auto-Map report without touching
+any agent logic: a new client-side run-visualizer renders the loop from the
+artifacts the engine already produces, and the two artifacts that were
+persisted-to-session-only are now also surfaced on the polled task result.
+
+## Changes
+
+1. `src/back/objects/mapping/Mapping.py` — `run_auto_assign_task` (batch) and
+   `run_single_auto_assign_task` (single): added `source_model`,
+   `mapping_evaluations`, and `mapping_run_log` to the `complete_task` result
+   dict. These were already accumulated and persisted to the session via
+   `save_mappings_to_session`; they are now also returned on the polled task so
+   the UI can render the loop from `GET /tasks/{id}` with no extra round-trip.
+   `pge_scorecard` was already present (computed by `score_mapping_run`).
+2. `src/front/static/mapping/js/mapping-pge-visualizer.js` — NEW. `PgeVisualizer`
+   module: renders the verdict pill + KPI strip (entity/relationship
+   completeness, id-integrity, sql-exec failures, coverage-loss), the three
+   gate tiers (absolute / ratio / regression), a per-item loop trace
+   (Generator › Evaluator › Critic chain with attempt numbers, retry hints, and
+   re-plan markers) enriched with per-item eval metrics, and a collapsible
+   planner source-model panel (table→class candidates with confidence bars,
+   canonical ids, join keys with overlap, planner skips). Fully defensive — any
+   field may be missing; renders nothing if there is no PGE payload.
+3. `src/front/static/mapping/css/mapping-pge-visualizer.css` — NEW. Component
+   styles (ob-pge-* classes) for the card, KPI strip, gate chips, loop-trace
+   timeline, and source-model tables. Bootstrap 5.3 for everything else.
+4. `src/front/static/mapping/js/mapping-autoassign.js` — stash `this.taskResult`
+   in both completion paths (live monitor + resumed task) and call
+   `PgeVisualizer.render(...)` from `showReport()`; clear it in `reset()`.
+5. `src/front/templates/partials/mapping/_mapping_autoassign.html` — added the
+   `#autoAssignPgeVisualizer` mount container inside the report section.
+6. `src/front/templates/mapping.html` — included the new CSS + JS assets.
+
+## Modified / new files
+
+- src/back/objects/mapping/Mapping.py (modified)
+- src/front/static/mapping/js/mapping-pge-visualizer.js (new)
+- src/front/static/mapping/css/mapping-pge-visualizer.css (new)
+- src/front/static/mapping/js/mapping-autoassign.js (modified)
+- src/front/templates/partials/mapping/_mapping_autoassign.html (modified)
+- src/front/templates/mapping.html (modified)
+
+## Tests
+
+- `pytest tests/agents/agent_mapping_pge/` — 82 passed, 1 warning (no regression
+  from the task-result additions; the engine/contracts are untouched).
+- `node --check` passes on both modified/new JS modules.
+- `python -m py_compile` passes on Mapping.py.
+- Visual verification: rendered the visualizer against a realistic synthetic
+  task result via headless Chrome (file:// harness). Confirmed: verdict pill,
+  KPI strip, three gate tiers, per-item loop chains (incl. a 2-attempt
+  null-id-hint retry and a re-plan escalation), per-item eval metrics, and the
+  planner source-model panel all render correctly. Screenshots captured.

changelogs/v0.5.0/FiifiB_2026-06-20.log

@@ -0,0 +1,42 @@
+# v0.5.0 — PGE Run-Visualizer: reasoning-wrap fix + live verification
+
+## Context
+
+Continuation of the PGE run-visualizer (see FiifiB_2026-06-19.log). Verified the
+visualizer end-to-end with a real, live PGE auto-map run on the deployed app
+(ontobricks-pgeviz). The live run surfaced one rendering issue not present in the
+synthetic-data tests: the semantic critic's `reasoning` (carried in
+`EvalReport.metrics.reasoning`) is long free text that rendered inline and
+overflowed the viewport.
+
+## Changes
+
+1. `src/front/static/mapping/js/mapping-pge-visualizer.js` — `renderItem()`:
+   split per-item eval metrics into short scalar metrics (rendered inline as
+   before) and long free-text fields >60 chars such as `reasoning` (now rendered
+   as a wrapped `.ob-pge-reasoning` block instead of a nowrap inline span).
+2. `src/front/static/mapping/css/mapping-pge-visualizer.css` — added
+   `.ob-pge-reasoning` (white-space: normal, overflow-wrap: anywhere) so long
+   critic reasoning wraps cleanly.
+
+## Modified files
+
+- src/front/static/mapping/js/mapping-pge-visualizer.js
+- src/front/static/mapping/css/mapping-pge-visualizer.css
+
+## Tests / verification
+
+- `node --check` passes on the visualizer JS.
+- LIVE end-to-end verification on deployed app `ontobricks-pgeviz`
+  (https://ontobricks-pgeviz-7474646666236453.aws.databricksapps.com):
+  loaded trust_a.maternity_episode metadata → applied a maternity ontology
+  (MaternityEpisode, Baby) → ran a real PGE auto-assign → both entities PASS in
+  1 attempt. The deployed PgeVisualizer rendered the real run: stage chips,
+  RED verdict, KPI strip (entity/rel/id-integrity 100%, 0 sql failures,
+  0% coverage-loss), gate tiers (Tier-1 absolute FAILED on orphan_class_count=2,
+  Tier-2/3 pass), per-item Generator→Evaluator→Critic chains with full wrapped
+  critic reasoning, per-item metrics, and the planner source-model (candidate
+  confidences 98%/90%, canonical-id analysis with real row counts 1228/776).
+  Zero console errors. Screenshot captured.
+- Deployment used the dev-lakebase target against a dedicated Lakebase project
+  (ontobricks-pgeviz-db) for full isolation from shared production.

docs/INFO.md

@@ -323,7 +323,7 @@ src/
 │   ├── llm_utils.py             # Shared LLM call with retry
 │   ├── tools/                   # Shared agent tools (context, metadata, SQL, …)
 │   ├── agent_owl_generator/     # OWL ontology generation agent
-│   ├── agent_auto_assignment/   # Entity/relationship → SQL mapping agent
+│   ├── agent_mapping_pge/       # Mapping PGE pipeline (Planner → Generator → Evaluator)
 │   ├── agent_auto_icon_assign/  # Emoji icon mapping agent
 │   └── agent_ontology_assistant/# Conversational assistant + ResponsesAgent wrapper
 │