Document canonical Eval Lab failure and workload boundaries

README.md

@@ -1,17 +1,39 @@
 # gdev-agent
 
-![Python 3.12](https://img.shields.io/badge/python-3.12-3776AB?logo=python&logoColor=white) ![FastAPI](https://img.shields.io/badge/fastapi-api-009688?logo=fastapi&logoColor=white) ![Postgres](https://img.shields.io/badge/postgres-pgvector-4169E1?logo=postgresql&logoColor=white) ![Docker Compose](https://img.shields.io/badge/docker-compose-2496ED?logo=docker&logoColor=white)
-
-`gdev-agent` is a governed, multi-tenant LLM workflow reliability system for
-game-studio support: it receives support webhooks, blocks unsafe input before
-any model call, classifies and extracts structured data with an LLM, routes
-risky actions into human approval, and records the resulting audit, cost, and
-analytics trail behind one HTTP API.
-
-Status: local evidence baseline complete. The current stack is pilot-grade:
-Docker Compose setup, synthetic demo and eval paths, and repository tests. This
-README does not claim production SaaS readiness, external deployment, or live
-customer usage.
+`gdev-agent` is a governed local support-triage reference workload for
+game-studio scenarios. It receives signed support webhooks, blocks unsafe input
+before any model call, classifies and extracts structured data, routes risky
+actions into human approval, and records an audit/cost trail behind one HTTP
+API.
+
+## Current Maturity
+
+The supported proof boundary is a tested local Docker Compose workload with
+synthetic demo/eval data and database-enforced tenant isolation. Its 55-case
+conformance set passes, while the harder published 100-case Eval Lab gate fails.
+There is no claimed pilot, external deployment, live customer traffic,
+production readiness, or production SLO.
+
+## Relationship to the Portfolio
+
+- This repository owns application behavior, signed ingress, approval/audit
+  flows, tenant controls, and candidate fixes.
+- [Eval Ground Truth Lab](https://github.com/ashishki/Eval-Ground-Truth-Lab)
+  owns external workflow-quality gates. Its canonical 100-case result for this
+  revision is FAIL and remains authoritative for those challenge outcomes.
+- [Agent Runtime Grid](https://github.com/ashishki/Agent-Runtime-Grid) is an
+  optional execution layer. It does not replace gdev lifecycle/security tests
+  or Eval Lab quality decisions.
+- [AI Workflow Playbook](https://github.com/ashishki/AI_workflow_playbook) is an
+  independent governance companion, not a runtime dependency.
+- The thin umbrella pins compatible revisions and runs integration evidence; it
+  does not absorb component code or history.
+
+## Product Boundary and Non-Goals
+
+The repository demonstrates one bounded local support workflow. It is not a
+hosted support platform, customer-data product, generic agent framework,
+production multi-tenant SaaS, or proof that the demo classifier generalizes.
 
 ## Evidence Path
 
@@ -26,18 +48,22 @@ For a claim-by-claim proof map, start with
 | Architecture and workflow boundaries | [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md), [docs/architecture-diagram.md](docs/architecture-diagram.md) | Implemented local stack with documented gaps and ADRs |
 | Agent harness boundary | [docs/HARNESS_CARD.md](docs/HARNESS_CARD.md), [docs/TRACE_SCHEMA.md](docs/TRACE_SCHEMA.md), [AGENTS.md](AGENTS.md) | Model + prompt/tool loop + guards + approvals + trace + eval are reviewed as one bounded harness |
 | Repeatable demo path | [docs/DEMO.md](docs/DEMO.md) | Local Compose demo with deterministic/free mode |
-| Evaluation discipline | [docs/EVALUATION.md](docs/EVALUATION.md), [docs/EVAL_REPORT.md](docs/EVAL_REPORT.md), [docs/EVAL_SCOPE_RECONCILIATION.md](docs/EVAL_SCOPE_RECONCILIATION.md) | 180-case internal smoke eval, 55-case Eval Lab conformance baseline, and a separately identified unexecuted 100-case challenge scope |
+| Evaluation discipline | [docs/EVALUATION.md](docs/EVALUATION.md), [docs/EVAL_REPORT.md](docs/EVAL_REPORT.md), [docs/EVAL_SCOPE_RECONCILIATION.md](docs/EVAL_SCOPE_RECONCILIATION.md) | 180-case internal smoke eval, 55-case Eval Lab conformance baseline, and a canonical 100-case challenge run whose stricter gate failed |
 | Observability | [docs/observability.md](docs/observability.md) | Metrics, traces, logs, and alerting design for local evidence |
 | Load profile | [docs/load-profile.md](docs/load-profile.md), [docs/LOAD_TEST_REPORT.md](docs/LOAD_TEST_REPORT.md) | Local deterministic/synthetic report and scenario targets; not production capacity claims |
 | Tenant isolation and security | [docs/TENANT_ISOLATION.md](docs/TENANT_ISOLATION.md), [docs/data-map.md#6-tenant-isolation-model](docs/data-map.md#6-tenant-isolation-model), [docs/ARCHITECTURE.md#7-security-model](docs/ARCHITECTURE.md#7-security-model) | RLS, tenant-scoped JWT, webhook signature, secrets, approval, and cost ledger boundaries |
 | Tests | [Current State](#current-state) | 2026-07-13 local baseline: 310 passing tests; rerun locally before relying on it |
 | Failure modes and SLO/runbook | [docs/FAILURE_MODES.md](docs/FAILURE_MODES.md), [docs/SLO_RUNBOOK.md](docs/SLO_RUNBOOK.md), [docs/observability.md#alert-runbooks](docs/observability.md#alert-runbooks) | Local taxonomy and runbook evidence; external incident evidence is out of scope |
 | Deployment readiness boundaries | [docs/DEPLOYMENT_READINESS.md](docs/DEPLOYMENT_READINESS.md), [Known Limits](#known-limits) | Secrets checklist, backup/restore notes, local production-like config, and known limitations without production readiness claims |
-| Known limits and production changes | [Known Limits](#known-limits), [docs/DEPLOYMENT_READINESS.md](docs/DEPLOYMENT_READINESS.md) | Explicitly bounded as pilot/local evidence, not production SaaS readiness |
+| Known limits and production changes | [Known Limits](#known-limits), [docs/DEPLOYMENT_READINESS.md](docs/DEPLOYMENT_READINESS.md) | Explicitly bounded as local reference evidence, not production SaaS readiness |
 
 ## Why This Project Exists
 
-Game studios deal with billing disputes, account-access incidents, bug reports, moderation signals, and repetitive gameplay questions at a volume where manual triage becomes slow and brittle. `gdev-agent` is the orchestration layer between inbound support traffic and downstream systems: it keeps routine requests moving, forces human review when confidence or risk is low, and preserves tenant isolation, observability, and cost controls.
+The synthetic reference scenario covers billing disputes, account-access
+incidents, bug reports, moderation signals, and repetitive gameplay questions.
+`gdev-agent` exercises the orchestration boundary between inbound support
+traffic and downstream tools: routine paths move through deterministic controls,
+while low-confidence or risky actions require review.
 
 ## Architecture
 
@@ -79,7 +105,7 @@ The current stack includes FastAPI, Redis, PostgreSQL with Row-Level Security, p
 This path is aligned to [docker-compose.yml](docker-compose.yml) and is the fastest way to get a healthy local stack.
 
 ```bash
-git clone https://github.com/your-handle/gdev-agent.git
+git clone https://github.com/ashishki/gdev-agent.git
 cd gdev-agent
 cp .env.example .env
 docker compose up --build
@@ -217,7 +243,7 @@ Most endpoints outside `/health`, `/webhook`, and `/metrics` require JWT auth pl
 
 - [docs/EVIDENCE_INDEX.md](docs/EVIDENCE_INDEX.md): evidence question map and claim-by-claim proof table.
 - [docs/STACK_OVERVIEW.md](docs/STACK_OVERVIEW.md): three-project stack map and provider strategy.
-- [docs/EVAL_SCOPE_RECONCILIATION.md](docs/EVAL_SCOPE_RECONCILIATION.md): reconciles the internal 180-case smoke, Eval Lab 55-case conformance baseline, unexecuted 100-case challenge scope, and Runtime Grid proofs.
+- [docs/EVAL_SCOPE_RECONCILIATION.md](docs/EVAL_SCOPE_RECONCILIATION.md): reconciles the internal 180-case smoke, Eval Lab 55-case conformance baseline, canonical 100-case challenge failure, and Runtime Grid proofs.
 - [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md): system structure, service boundaries, request flow, deployment view.
 - [docs/HARNESS_CARD.md](docs/HARNESS_CARD.md): agent harness boundary across model, tools, memory, retries, permissions, HITL, trace, and eval.
 - [docs/TRACE_SCHEMA.md](docs/TRACE_SCHEMA.md): trace completeness contract for debugging, eval, audit, and approval retrospectives.
@@ -237,35 +263,39 @@ Most endpoints outside `/health`, `/webhook`, and `/metrics` require JWT auth pl
 
 ## Known Limits
 
-- The project is pilot-grade/local evidence. It has no claimed external
+- The project is a tested local reference workload. It has no claimed external
   deployment, production SaaS readiness, live tenant traffic, or real customer
   operations.
 - Demo, eval, and load evidence is synthetic unless a later report explicitly
   says otherwise. The current load report is deterministic/local evidence, not
   live capacity proof.
 - Eval metrics have multiple scopes. The internal 180-case smoke report exposes
   broad demo-mode routing gaps, while the external Eval Lab 55-case baseline is
-  an integration/conformance pass over the configured `/webhook` adapter. The
-  separate 100-case challenge dataset has no canonical executed run yet. See
+  an integration/conformance pass over the configured `/webhook` adapter. A
+  canonical Eval Lab run of the separate 100-case challenge against exact
+  revision `0e4c5f0fd50382bbf12ffd35cfca4632384fb0cc` failed its stricter gate:
+  reconciled pass rate `0.32`, classification accuracy `0.244444`, 68 unexpected
+  failures, and 58 blocking failures. This is published negative evidence, not
+  a threshold pass. See
   [docs/EVAL_SCOPE_RECONCILIATION.md](docs/EVAL_SCOPE_RECONCILIATION.md).
 - Live load measurements remain out of scope for the current local evidence.
-  Deployment readiness notes are local/pilot-only and explicitly do not prove
+  Deployment readiness notes are local-only and explicitly do not prove
   production readiness.
 - Live LLM behavior requires a real Anthropic API key and budget controls; the
   local stack is the supported review path today.
 
 ## Current State
 
-The local stack is pilot-grade and feature-complete enough to demonstrate the
-governed request pipeline. It includes the multi-tenant storage foundation,
+The local stack demonstrates the governed request pipeline. It includes the
+tenant-scoped storage foundation,
 JWT/RBAC boundary, approval hardening, eval APIs with budget enforcement, auth
 service flows, embedding persistence, RCA clustering with persisted cluster
 membership, service-layer separation for the main write/auth/eval workflows
 with read-route extraction still tracked as architecture drift, Dockerized
-observability, admin CLI, and the n8n workflow artifacts needed for demo or
-pilot-style setups.
+observability, admin CLI, and n8n workflow artifacts for local demonstrations.
 
-The 2026-07-13 local baseline is **310 tests passed** (unit + integration,
+At revision `0e4c5f0fd50382bbf12ffd35cfca4632384fb0cc`, the dated 2026-07-13
+local baseline is **310 tests passed** (unit + integration,
 including migration up/down, role flags, FORCE RLS topology, cross-tenant
 rejection, eval metric validators, load fixtures, observability signals, and
 cluster membership persistence). The same repair run also passed the 180-case

docs/CASE_STUDY.md

@@ -73,17 +73,22 @@ Latest committed eval results from [docs/EVAL_REPORT.md](EVAL_REPORT.md):
 | Invalid structured output rate | 0.0000 | Structured output contract holds in demo mode. |
 | Classification accuracy | 0.1698 | Observed only; demo classifier does not claim broad taxonomy quality. |
 
-The CI eval regression gate is active for smoke regressions. Stricter quality
-gates remain future work.
+The CI eval regression gate is active for smoke regressions. A separate,
+stricter Eval Lab challenge gate now exists and the canonical run failed it.
 
 There is also an external Eval Lab integration baseline over 55 curated
 gdev-agent triage cases. That baseline calls a live local `gdev-agent` through
 the configured `/webhook` adapter and currently records 55 cases, zero adapter
 errors, and zero deterministic validator failures. This does not contradict the
 weaker internal 180-case smoke metrics: the two reports have different scopes.
-Eval Lab also contains a 100-case diagnostic challenge dataset, but no canonical
-executed challenge run; it must not be read as a `100/100` result.
-See [docs/EVAL_SCOPE_RECONCILIATION.md](EVAL_SCOPE_RECONCILIATION.md).
+Eval Lab also ran its 100-case diagnostic challenge against exact revision
+`0e4c5f0fd50382bbf12ffd35cfca4632384fb0cc`: 90 HTTP candidate cases plus 10
+deterministic provider-fault injections. The canonical gate **failed** with a
+`0.32` reconciled pass rate, `0.244444` classification accuracy, 68 unexpected
+failures, and 58 blocking failures. The provider-fault slice matched `10/10`,
+but that is harness evidence, not observed provider reliability. See the
+[content-addressed Eval Lab package](https://github.com/ashishki/Eval-Ground-Truth-Lab/tree/main/docs/evidence/releases/v0.2.0/gdev-agent-challenge)
+and [scope reconciliation](EVAL_SCOPE_RECONCILIATION.md).
 
 ## Load Results
 
@@ -116,6 +121,9 @@ metrics, and test/evidence pointers.
   proven by the committed baseline.
 - The clean 55-case Eval Lab baseline proves the current integration contract,
   not broad production triage quality across every internal smoke taxonomy.
+- The failed 100-case challenge is a public diagnostic set, not a blind or
+  real-customer benchmark; improving it must not mean copying case phrases into
+  routing rules or weakening the frozen thresholds.
 - Redis stores ephemeral coordination state; Postgres is the durable source of
   record.
 - Read APIs still have known service-extraction debt for ticket, analytics, and

docs/EVALUATION.md

@@ -2,8 +2,8 @@
 
 ## 1. Overview
 
-The eval subsystem runs a tenant-scoped offline quality check against the same agent flow used by
-production triage. `app/routers/eval.py` exposes the HTTP entrypoints, `app/services/eval_service.py`
+The eval subsystem runs a tenant-scoped offline quality check against the same implemented agent
+flow used by the triage API. `app/routers/eval.py` exposes the HTTP entrypoints, `app/services/eval_service.py`
 queues and tracks runs in `eval_runs`, and `eval/runner.py` executes the dataset and writes the
 resulting metrics back to Postgres.
 
@@ -215,3 +215,31 @@ Operational learning metrics:
 - Rejections are tracked separately from explicit overrides. A rejected action is an override, but
   an approved action with `corrected_category`, `corrected_urgency`, `corrected_action_tool`, or
   `override_reason` is also counted as override feedback.
+
+## 8. Canonical Eval Lab Challenge Evidence
+
+Eval Ground Truth Lab owns a separate public 100-case diagnostic challenge. Its
+canonical 2026-07-13 run fixed the gdev-agent candidate at
+`0e4c5f0fd50382bbf12ffd35cfca4632384fb0cc` with image digest
+`sha256:7dc9fef2ec6fe25745405546ec69f6a6f64c1bfa9f052dc54abfd65498a6f6da`.
+The run made 90 actual HTTP candidate calls and reconciled 10 deterministic
+provider-fault injections. Redis started clean, and Eval Lab namespaced both
+`request_id` and `message_id` as
+`gdev-eval-v1-5c65a837141710c3f31f9978823394bd6d51feb3889524dd1ca67bbcf27c4222`
+so results from another candidate/run could not satisfy dedup lookups.
+
+The challenge gate **failed**. Reconciled pass rate was `0.32`, classification
+accuracy was `0.244444`, 68 cases were unexpected failures, 58 were blocking
+failures, human review was observed in 46 cases, and human-escalation recall was
+`0.46`. Expected-failure match was `1.0` for the injected provider-fault slice;
+unsafe auto-approval rate, invalid structured-output rate, and cost per case
+were each `0`. Local p95 latency was `890.379885 ms`.
+
+This is synthetic/local negative evidence. It does not prove production quality,
+real provider reliability, customer usage, or a passed quality target. The
+[verified Eval Lab package](https://github.com/ashishki/Eval-Ground-Truth-Lab/tree/main/docs/evidence/releases/v0.2.0/gdev-agent-challenge)
+is content-addressed as
+`sha256:656face21f27b496d4d3e8bb0b588824f5737d122c1275c710f3e5b15ff94b4b`.
+The package records the failed blocking-count, accuracy, human-review,
+human-escalation, and unexpected-failure thresholds and remains the authority
+for per-case outcomes.

docs/EVAL_REPORT.md

@@ -20,8 +20,9 @@ For cross-project interpretation, read this report with
 internal eval is a broad smoke/gap-discovery surface. The separate Eval Ground
 Truth Lab 55-case baseline is a curated live local integration/conformance eval
 over the configured `/webhook` adapter. Eval Lab also contains a separate
-100-case diagnostic challenge dataset that has not been recorded as a canonical
-executed baseline; see the reconciliation document before comparing counts.
+100-case diagnostic challenge with a canonical failed run against this exact
+gdev-agent revision; see the reconciliation document before comparing counts or
+interpreting the separate gates.
 
 ## Environment Assumptions
 
@@ -92,9 +93,15 @@ current adapter/conformance contract for the curated Eval Lab dataset.
 
 Eval Lab's 100-case `challenge_v1.jsonl` is an executable diagnostic surface
 with a manifest, threshold gate, and explicit deterministic fault injection for
-the final provider-error slice. It is still not a canonical external-system run
-until that command records a fixed gdev-agent revision and verifies its evidence
-manifest.
+the final provider-error slice. Its canonical local run fixed gdev-agent at
+`0e4c5f0fd50382bbf12ffd35cfca4632384fb0cc`, executed 90 HTTP candidate cases,
+and reconciled 10 deterministic fault injections. The gate **failed**: `0.32`
+reconciled pass rate, `0.244444` classification accuracy, 68 unexpected failures,
+58 blocking failures, 46 human-review outcomes, and `0.46` human-escalation
+recall. The verified package is
+[published by Eval Lab](https://github.com/ashishki/Eval-Ground-Truth-Lab/tree/main/docs/evidence/releases/v0.2.0/gdev-agent-challenge)
+with content address
+`sha256:656face21f27b496d4d3e8bb0b588824f5737d122c1275c710f3e5b15ff94b4b`.
 
 It does not invalidate this internal 180-case report. This report remains the
 broader local smoke taxonomy and intentionally keeps weak routing and