From 6658da69b8d0b1e1a00648ddbc95b85036579747 Mon Sep 17 00:00:00 2001 From: Artem Shishkin Date: Mon, 13 Jul 2026 17:45:01 +0400 Subject: [PATCH 1/2] docs: publish canonical Eval Lab challenge failure --- README.md | 17 ++++--- docs/CASE_STUDY.md | 18 +++++-- docs/EVALUATION.md | 32 +++++++++++- docs/EVAL_REPORT.md | 17 +++++-- docs/EVAL_SCOPE_RECONCILIATION.md | 49 +++++++++++++++---- docs/EVIDENCE_INDEX.md | 4 +- docs/STACK_OVERVIEW.md | 2 +- .../GDEV_P0_TRUTH_REPAIR_2026-07-13.md | 15 +++--- tests/test_harness_docs.py | 5 ++ 9 files changed, 122 insertions(+), 37 deletions(-) diff --git a/README.md b/README.md index e70e870..4145e87 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,5 @@ # gdev-agent -![Python 3.12](https://img.shields.io/badge/python-3.12-3776AB?logo=python&logoColor=white) ![FastAPI](https://img.shields.io/badge/fastapi-api-009688?logo=fastapi&logoColor=white) ![Postgres](https://img.shields.io/badge/postgres-pgvector-4169E1?logo=postgresql&logoColor=white) ![Docker Compose](https://img.shields.io/badge/docker-compose-2496ED?logo=docker&logoColor=white) - `gdev-agent` is a governed, multi-tenant LLM workflow reliability system for game-studio support: it receives support webhooks, blocks unsafe input before any model call, classifies and extracts structured data with an LLM, routes @@ -26,7 +24,7 @@ For a claim-by-claim proof map, start with | Architecture and workflow boundaries | [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md), [docs/architecture-diagram.md](docs/architecture-diagram.md) | Implemented local stack with documented gaps and ADRs | | Agent harness boundary | [docs/HARNESS_CARD.md](docs/HARNESS_CARD.md), [docs/TRACE_SCHEMA.md](docs/TRACE_SCHEMA.md), [AGENTS.md](AGENTS.md) | Model + prompt/tool loop + guards + approvals + trace + eval are reviewed as one bounded harness | | Repeatable demo path | [docs/DEMO.md](docs/DEMO.md) | Local Compose demo with deterministic/free mode | -| Evaluation discipline | [docs/EVALUATION.md](docs/EVALUATION.md), [docs/EVAL_REPORT.md](docs/EVAL_REPORT.md), [docs/EVAL_SCOPE_RECONCILIATION.md](docs/EVAL_SCOPE_RECONCILIATION.md) | 180-case internal smoke eval, 55-case Eval Lab conformance baseline, and a separately identified unexecuted 100-case challenge scope | +| Evaluation discipline | [docs/EVALUATION.md](docs/EVALUATION.md), [docs/EVAL_REPORT.md](docs/EVAL_REPORT.md), [docs/EVAL_SCOPE_RECONCILIATION.md](docs/EVAL_SCOPE_RECONCILIATION.md) | 180-case internal smoke eval, 55-case Eval Lab conformance baseline, and a canonical 100-case challenge run whose stricter gate failed | | Observability | [docs/observability.md](docs/observability.md) | Metrics, traces, logs, and alerting design for local evidence | | Load profile | [docs/load-profile.md](docs/load-profile.md), [docs/LOAD_TEST_REPORT.md](docs/LOAD_TEST_REPORT.md) | Local deterministic/synthetic report and scenario targets; not production capacity claims | | Tenant isolation and security | [docs/TENANT_ISOLATION.md](docs/TENANT_ISOLATION.md), [docs/data-map.md#6-tenant-isolation-model](docs/data-map.md#6-tenant-isolation-model), [docs/ARCHITECTURE.md#7-security-model](docs/ARCHITECTURE.md#7-security-model) | RLS, tenant-scoped JWT, webhook signature, secrets, approval, and cost ledger boundaries | @@ -217,7 +215,7 @@ Most endpoints outside `/health`, `/webhook`, and `/metrics` require JWT auth pl - [docs/EVIDENCE_INDEX.md](docs/EVIDENCE_INDEX.md): evidence question map and claim-by-claim proof table. - [docs/STACK_OVERVIEW.md](docs/STACK_OVERVIEW.md): three-project stack map and provider strategy. -- [docs/EVAL_SCOPE_RECONCILIATION.md](docs/EVAL_SCOPE_RECONCILIATION.md): reconciles the internal 180-case smoke, Eval Lab 55-case conformance baseline, unexecuted 100-case challenge scope, and Runtime Grid proofs. +- [docs/EVAL_SCOPE_RECONCILIATION.md](docs/EVAL_SCOPE_RECONCILIATION.md): reconciles the internal 180-case smoke, Eval Lab 55-case conformance baseline, canonical 100-case challenge failure, and Runtime Grid proofs. - [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md): system structure, service boundaries, request flow, deployment view. - [docs/HARNESS_CARD.md](docs/HARNESS_CARD.md): agent harness boundary across model, tools, memory, retries, permissions, HITL, trace, and eval. - [docs/TRACE_SCHEMA.md](docs/TRACE_SCHEMA.md): trace completeness contract for debugging, eval, audit, and approval retrospectives. @@ -245,8 +243,12 @@ Most endpoints outside `/health`, `/webhook`, and `/metrics` require JWT auth pl live capacity proof. - Eval metrics have multiple scopes. The internal 180-case smoke report exposes broad demo-mode routing gaps, while the external Eval Lab 55-case baseline is - an integration/conformance pass over the configured `/webhook` adapter. The - separate 100-case challenge dataset has no canonical executed run yet. See + an integration/conformance pass over the configured `/webhook` adapter. A + canonical Eval Lab run of the separate 100-case challenge against exact + revision `0e4c5f0fd50382bbf12ffd35cfca4632384fb0cc` failed its stricter gate: + reconciled pass rate `0.32`, classification accuracy `0.244444`, 68 unexpected + failures, and 58 blocking failures. This is published negative evidence, not + a threshold pass. See [docs/EVAL_SCOPE_RECONCILIATION.md](docs/EVAL_SCOPE_RECONCILIATION.md). - Live load measurements remain out of scope for the current local evidence. Deployment readiness notes are local/pilot-only and explicitly do not prove @@ -265,7 +267,8 @@ with read-route extraction still tracked as architecture drift, Dockerized observability, admin CLI, and the n8n workflow artifacts needed for demo or pilot-style setups. -The 2026-07-13 local baseline is **310 tests passed** (unit + integration, +At revision `0e4c5f0fd50382bbf12ffd35cfca4632384fb0cc`, the dated 2026-07-13 +local baseline is **310 tests passed** (unit + integration, including migration up/down, role flags, FORCE RLS topology, cross-tenant rejection, eval metric validators, load fixtures, observability signals, and cluster membership persistence). The same repair run also passed the 180-case diff --git a/docs/CASE_STUDY.md b/docs/CASE_STUDY.md index cba9507..9759fbf 100644 --- a/docs/CASE_STUDY.md +++ b/docs/CASE_STUDY.md @@ -73,17 +73,22 @@ Latest committed eval results from [docs/EVAL_REPORT.md](EVAL_REPORT.md): | Invalid structured output rate | 0.0000 | Structured output contract holds in demo mode. | | Classification accuracy | 0.1698 | Observed only; demo classifier does not claim broad taxonomy quality. | -The CI eval regression gate is active for smoke regressions. Stricter quality -gates remain future work. +The CI eval regression gate is active for smoke regressions. A separate, +stricter Eval Lab challenge gate now exists and the canonical run failed it. There is also an external Eval Lab integration baseline over 55 curated gdev-agent triage cases. That baseline calls a live local `gdev-agent` through the configured `/webhook` adapter and currently records 55 cases, zero adapter errors, and zero deterministic validator failures. This does not contradict the weaker internal 180-case smoke metrics: the two reports have different scopes. -Eval Lab also contains a 100-case diagnostic challenge dataset, but no canonical -executed challenge run; it must not be read as a `100/100` result. -See [docs/EVAL_SCOPE_RECONCILIATION.md](EVAL_SCOPE_RECONCILIATION.md). +Eval Lab also ran its 100-case diagnostic challenge against exact revision +`0e4c5f0fd50382bbf12ffd35cfca4632384fb0cc`: 90 HTTP candidate cases plus 10 +deterministic provider-fault injections. The canonical gate **failed** with a +`0.32` reconciled pass rate, `0.244444` classification accuracy, 68 unexpected +failures, and 58 blocking failures. The provider-fault slice matched `10/10`, +but that is harness evidence, not observed provider reliability. See the +[content-addressed Eval Lab package](https://github.com/ashishki/Eval-Ground-Truth-Lab/tree/main/docs/evidence/releases/v0.2.0/gdev-agent-challenge) +and [scope reconciliation](EVAL_SCOPE_RECONCILIATION.md). ## Load Results @@ -116,6 +121,9 @@ metrics, and test/evidence pointers. proven by the committed baseline. - The clean 55-case Eval Lab baseline proves the current integration contract, not broad production triage quality across every internal smoke taxonomy. +- The failed 100-case challenge is a public diagnostic set, not a blind or + real-customer benchmark; improving it must not mean copying case phrases into + routing rules or weakening the frozen thresholds. - Redis stores ephemeral coordination state; Postgres is the durable source of record. - Read APIs still have known service-extraction debt for ticket, analytics, and diff --git a/docs/EVALUATION.md b/docs/EVALUATION.md index 2fd9e3d..f4f38b0 100644 --- a/docs/EVALUATION.md +++ b/docs/EVALUATION.md @@ -2,8 +2,8 @@ ## 1. Overview -The eval subsystem runs a tenant-scoped offline quality check against the same agent flow used by -production triage. `app/routers/eval.py` exposes the HTTP entrypoints, `app/services/eval_service.py` +The eval subsystem runs a tenant-scoped offline quality check against the same implemented agent +flow used by the triage API. `app/routers/eval.py` exposes the HTTP entrypoints, `app/services/eval_service.py` queues and tracks runs in `eval_runs`, and `eval/runner.py` executes the dataset and writes the resulting metrics back to Postgres. @@ -215,3 +215,31 @@ Operational learning metrics: - Rejections are tracked separately from explicit overrides. A rejected action is an override, but an approved action with `corrected_category`, `corrected_urgency`, `corrected_action_tool`, or `override_reason` is also counted as override feedback. + +## 8. Canonical Eval Lab Challenge Evidence + +Eval Ground Truth Lab owns a separate public 100-case diagnostic challenge. Its +canonical 2026-07-13 run fixed the gdev-agent candidate at +`0e4c5f0fd50382bbf12ffd35cfca4632384fb0cc` with image digest +`sha256:7dc9fef2ec6fe25745405546ec69f6a6f64c1bfa9f052dc54abfd65498a6f6da`. +The run made 90 actual HTTP candidate calls and reconciled 10 deterministic +provider-fault injections. Redis started clean, and Eval Lab namespaced both +`request_id` and `message_id` as +`gdev-eval-v1-5c65a837141710c3f31f9978823394bd6d51feb3889524dd1ca67bbcf27c4222` +so results from another candidate/run could not satisfy dedup lookups. + +The challenge gate **failed**. Reconciled pass rate was `0.32`, classification +accuracy was `0.244444`, 68 cases were unexpected failures, 58 were blocking +failures, human review was observed in 46 cases, and human-escalation recall was +`0.46`. Expected-failure match was `1.0` for the injected provider-fault slice; +unsafe auto-approval rate, invalid structured-output rate, and cost per case +were each `0`. Local p95 latency was `890.379885 ms`. + +This is synthetic/local negative evidence. It does not prove production quality, +real provider reliability, customer usage, or a passed quality target. The +[verified Eval Lab package](https://github.com/ashishki/Eval-Ground-Truth-Lab/tree/main/docs/evidence/releases/v0.2.0/gdev-agent-challenge) +is content-addressed as +`sha256:656face21f27b496d4d3e8bb0b588824f5737d122c1275c710f3e5b15ff94b4b`. +The package records the failed blocking-count, accuracy, human-review, +human-escalation, and unexpected-failure thresholds and remains the authority +for per-case outcomes. diff --git a/docs/EVAL_REPORT.md b/docs/EVAL_REPORT.md index aff4031..707c41f 100644 --- a/docs/EVAL_REPORT.md +++ b/docs/EVAL_REPORT.md @@ -20,8 +20,9 @@ For cross-project interpretation, read this report with internal eval is a broad smoke/gap-discovery surface. The separate Eval Ground Truth Lab 55-case baseline is a curated live local integration/conformance eval over the configured `/webhook` adapter. Eval Lab also contains a separate -100-case diagnostic challenge dataset that has not been recorded as a canonical -executed baseline; see the reconciliation document before comparing counts. +100-case diagnostic challenge with a canonical failed run against this exact +gdev-agent revision; see the reconciliation document before comparing counts or +interpreting the separate gates. ## Environment Assumptions @@ -92,9 +93,15 @@ current adapter/conformance contract for the curated Eval Lab dataset. Eval Lab's 100-case `challenge_v1.jsonl` is an executable diagnostic surface with a manifest, threshold gate, and explicit deterministic fault injection for -the final provider-error slice. It is still not a canonical external-system run -until that command records a fixed gdev-agent revision and verifies its evidence -manifest. +the final provider-error slice. Its canonical local run fixed gdev-agent at +`0e4c5f0fd50382bbf12ffd35cfca4632384fb0cc`, executed 90 HTTP candidate cases, +and reconciled 10 deterministic fault injections. The gate **failed**: `0.32` +reconciled pass rate, `0.244444` classification accuracy, 68 unexpected failures, +58 blocking failures, 46 human-review outcomes, and `0.46` human-escalation +recall. The verified package is +[published by Eval Lab](https://github.com/ashishki/Eval-Ground-Truth-Lab/tree/main/docs/evidence/releases/v0.2.0/gdev-agent-challenge) +with content address +`sha256:656face21f27b496d4d3e8bb0b588824f5737d122c1275c710f3e5b15ff94b4b`. It does not invalidate this internal 180-case report. This report remains the broader local smoke taxonomy and intentionally keeps weak routing and diff --git a/docs/EVAL_SCOPE_RECONCILIATION.md b/docs/EVAL_SCOPE_RECONCILIATION.md index 3008d5f..33cf15b 100644 --- a/docs/EVAL_SCOPE_RECONCILIATION.md +++ b/docs/EVAL_SCOPE_RECONCILIATION.md @@ -9,7 +9,7 @@ answer different questions. | --- | --- | ---: | --- | --- | | Internal gdev-agent smoke eval | `eval/cases.jsonl`, `docs/EVAL_REPORT.md` | 180 | Does the local demo-mode workflow expose broad taxonomy, guard, routing, and unsafe-auto-approval regressions? | Broad smoke signal. It intentionally exposes known demo-policy quality gaps. | | Eval Lab integration baseline | `Eval-Ground-Truth-Lab/datasets/gdev_agent/triage_v1.jsonl`, `Eval-Ground-Truth-Lab/reports/gdev-agent/baseline_report.md` | 55 | Does Eval Lab's configured HTTP adapter reach a live local gdev-agent and validate the agreed triage contract? | Passing integration/conformance baseline: 55 cases, zero adapter errors, zero validator failures. | -| Eval Lab challenge diagnostic | `Eval-Ground-Truth-Lab/datasets/gdev_agent/challenge_v1.jsonl`, `Eval-Ground-Truth-Lab/datasets/gdev_agent/challenge_manifest.json`, `Eval-Ground-Truth-Lab/reports/gdev-agent/challenge_report.md` | 100 | Where should ambiguous, policy-stress, malformed, and provider-failure cases expose gaps? | The executable challenge command reconciles 90 candidate calls and 10 labeled harness fault injections. There is no canonical external-system run artifact yet. | +| Eval Lab challenge diagnostic | [Eval Lab v0.2.0 canonical challenge evidence](https://github.com/ashishki/Eval-Ground-Truth-Lab/tree/main/docs/evidence/releases/v0.2.0/gdev-agent-challenge) | 100 | Where do ambiguous, policy-stress, malformed, and provider-failure cases expose gaps in a fixed candidate? | Canonical local run against exact gdev-agent revision `0e4c5f0fd50382bbf12ffd35cfca4632384fb0cc`: 90 HTTP candidate calls plus 10 labeled harness fault injections. Gate **FAIL**. | | Runtime Grid artifact proof | `Agent-Runtime-Grid` `proof full-stack` | 20 default | Can selected Eval Lab/gdev evidence be run as queue-backed jobs with runtime artifacts, lifecycle state, and report cross-links? | Default runtime reliability proof over ready artifacts, not a live HTTP gdev-agent quality eval. | | Runtime Grid live-local proof | `Agent-Runtime-Grid` `proof full-stack-live-local` | operator-selected; 20 in latest snapshot | Can Grid workers call a local gdev-agent HTTP endpoint while preserving queue lifecycle, sanitized artifacts, and report links? | Optional local HTTP proof. The 2026-06-15 committed snapshot completed 20/20 queued jobs against local demo-mode gdev-agent, but it does not replace Eval Lab's quality report or claim production traffic. | @@ -33,10 +33,38 @@ So `55/55` in Eval Lab does not erase weak routing metrics in the broader internal report. It means the integration contract is passing for the current conformance set. -Likewise, the existence of 100 challenge cases is not a `100/100` result. Eval -Lab now represents its expected-failure slice explicitly, but a canonical score -still requires the fixed external gdev-agent revision to be run and recorded. -This repository therefore does not infer a pass rate from unit-test fixtures. +Likewise, the 100 challenge cases are not a `100/100` result. Eval Lab executed +the canonical run against the fixed revision above and published the failure. +The reconciled pass rate was `0.32`; 68 cases were unexpected failures, including +58 blocking failures. The 10 deterministic provider-fault cases matched their +expected failures, but they are harness evidence rather than observed candidate +outages. The full artifact remains the authority for the case-level results. + +## Canonical Challenge Result + +The 2026-07-13 run used a clean local Compose/Redis state and the image digest +`sha256:7dc9fef2ec6fe25745405546ec69f6a6f64c1bfa9f052dc54abfd65498a6f6da`. +Eval Lab applied request namespace +`gdev-eval-v1-5c65a837141710c3f31f9978823394bd6d51feb3889524dd1ca67bbcf27c4222` +to both `request_id` and `message_id`, preventing a prior candidate/run from +satisfying Redis dedup for this run. + +| Signal | Observed | Interpretation | +| --- | ---: | --- | +| Reconciled pass rate | `0.32` | Challenge gate failed; this is not a quality pass. | +| Classification accuracy | `0.244444` | Failed the `>= 0.70` challenge threshold. | +| Expected-failure match | `1.0` | All 10 deterministic provider-fault injections matched; not evidence of real provider outages. | +| Unexpected / blocking failures | `68` / `58` | Both failure-count gates failed. | +| Human review observed / escalation recall | `46` / `0.46` | Both human-routing gates failed. | +| Unsafe auto-approval / invalid output / cost per case | `0` / `0` / `$0` | These individual gates passed and do not override the overall failure. | +| Local p95 latency | `890.379885 ms` | Passed the challenge latency bound; not a production SLO. | + +The verified evidence package is content-addressed as +`sha256:656face21f27b496d4d3e8bb0b588824f5737d122c1275c710f3e5b15ff94b4b`. +Its failed thresholds are blocking-failure maximum, classification-accuracy +minimum, human-review-count minimum, human-escalation-recall minimum, and +unexpected-failure maximum. Labels and thresholds were not relaxed to turn the +run green. ## Smoke Gates vs Quality Targets @@ -53,8 +81,11 @@ This repository therefore does not infer a pass rate from unit-test fixtures. - Keep the 180-case internal eval as a broad smoke and gap-discovery surface. - Add stricter quality gates only when the demo/live policy is improved across the broad taxonomy. -- Run Eval Lab's first-class expected-failure/fault-injection command against a - fixed clean gdev-agent revision and publish its verified evidence manifest. +- Use the published case-level failures to improve general policy behavior; + do not copy challenge wording into rules or weaken labels/thresholds to make + the frozen public set pass. +- Re-run each candidate with a distinct deterministic request namespace and a + clean service state, then publish failures as well as passes. - Keep Runtime Grid `proof full-stack` as the reproducible artifact-linked proof, and use `proof full-stack-live-local` only as explicit local HTTP evidence when the operator has a local gdev-agent stack running. @@ -63,8 +94,8 @@ This repository therefore does not infer a pass rate from unit-test fixtures. Use the Eval Lab 55-case report to inspect integration correctness. Use the internal 180-case report to inspect known quality gaps and regression visibility. -Use the 100-case challenge assets to inspect planned diagnostic coverage, not as -an executed score until a canonical run artifact exists. +Use the [canonical 100-case challenge evidence](https://github.com/ashishki/Eval-Ground-Truth-Lab/tree/main/docs/evidence/releases/v0.2.0/gdev-agent-challenge) +to inspect the stricter gate failure and its case-level diagnostic coverage. Use Runtime Grid artifact evidence to inspect batch execution reliability, and use Runtime Grid live-local evidence only when you want to inspect queued local HTTP execution against gdev-agent. diff --git a/docs/EVIDENCE_INDEX.md b/docs/EVIDENCE_INDEX.md index c14cd5e..af3947f 100644 --- a/docs/EVIDENCE_INDEX.md +++ b/docs/EVIDENCE_INDEX.md @@ -14,7 +14,7 @@ production SaaS readiness. | architecture | [docs/architecture-diagram.md](architecture-diagram.md), [docs/ARCHITECTURE.md](ARCHITECTURE.md) | Implemented local stack and request flow; no external deployment claim. | | harness boundary | [docs/HARNESS_CARD.md](HARNESS_CARD.md), [docs/TRACE_SCHEMA.md](TRACE_SCHEMA.md), [../AGENTS.md](../AGENTS.md) | Model, prompt/tool loop, tools, memory, retries, permissions, human handoff, trace, and eval are reviewed as one bounded harness. | | control boundaries | [docs/CASE_STUDY.md#control-boundaries](CASE_STUDY.md#control-boundaries), [docs/TENANT_ISOLATION.md](TENANT_ISOLATION.md) | RLS, JWT/RBAC, HMAC, approval, output guard, cost, and observability controls. | -| quality evaluation | [docs/EVALUATION.md](EVALUATION.md), [docs/EVAL_REPORT.md](EVAL_REPORT.md), [docs/EVAL_SCOPE_RECONCILIATION.md](EVAL_SCOPE_RECONCILIATION.md) | 180-case internal smoke, 55-case Eval Lab conformance baseline, and a distinct unexecuted 100-case diagnostic challenge scope. | +| quality evaluation | [docs/EVALUATION.md](EVALUATION.md), [docs/EVAL_REPORT.md](EVAL_REPORT.md), [docs/EVAL_SCOPE_RECONCILIATION.md](EVAL_SCOPE_RECONCILIATION.md) | 180-case internal smoke, 55-case Eval Lab conformance baseline, and a canonical 100-case diagnostic challenge whose stricter gate failed. | | failure behavior | [docs/FAILURE_MODES.md](FAILURE_MODES.md), [docs/SLO_RUNBOOK.md](SLO_RUNBOOK.md) | Local taxonomy and runbook targets; no external incident evidence. | | baseline metrics | [README.md#current-state](../README.md#current-state), [docs/evidence/GDEV_P0_TRUTH_REPAIR_2026-07-13.md](evidence/GDEV_P0_TRUTH_REPAIR_2026-07-13.md), [docs/LOAD_TEST_REPORT.md](LOAD_TEST_REPORT.md), [docs/EVAL_REPORT.md](EVAL_REPORT.md) | 310-test 2026-07-13 local baseline, eval baseline, and local deterministic load fixture. | | demo path | [docs/DEMO.md](DEMO.md), [docs/evidence/GDEV_P0_TRUTH_REPAIR_2026-07-13.md](evidence/GDEV_P0_TRUTH_REPAIR_2026-07-13.md) | Deterministic Compose auth/approval demo output is recorded; no committed video/GIF artifact yet. | @@ -28,7 +28,7 @@ production SaaS readiness. | The system has a governed, multi-tenant LLM workflow architecture for support intake, triage, approval, and audit. | [docs/ARCHITECTURE.md](ARCHITECTURE.md), [docs/architecture-diagram.md](architecture-diagram.md), [README.md](../README.md) | `rg -n "WebhookService|ApprovalService|OutputGuard|Row-Level Security" docs/ARCHITECTURE.md README.md` | Architecture proof is current repo evidence, not an external deployment review. | | The agent is documented as a measurable harness, not just a model call. | [docs/HARNESS_CARD.md](HARNESS_CARD.md), [docs/TRACE_SCHEMA.md](TRACE_SCHEMA.md), [../AGENTS.md](../AGENTS.md), [../eval/harness_regression.jsonl](../eval/harness_regression.jsonl), [../tests/test_harness_docs.py](../tests/test_harness_docs.py) | `pytest tests/test_harness_docs.py -q` and `rg -n "Harness Boundary|Trace Requirements|No Silent Workaround" docs/HARNESS_CARD.md docs/TRACE_SCHEMA.md AGENTS.md` | The fixture is trace-oriented and synthetic; it is not yet wired into the main eval runner adapter. | | The demo path can exercise the local approval workflow. | [docs/DEMO.md](DEMO.md), [scripts/demo.py](../scripts/demo.py), [scripts/demo.sh](../scripts/demo.sh), [docs/evidence/GDEV_P0_TRUTH_REPAIR_2026-07-13.md](evidence/GDEV_P0_TRUTH_REPAIR_2026-07-13.md) | `docker compose exec -T agent python scripts/demo.py --url http://localhost:8000 --llm-mode demo` | Demo evidence is deterministic and local-only; it is not an external deployment proof. | -| Eval exists as a repeatable quality signal. | [docs/EVALUATION.md](EVALUATION.md), [docs/EVAL_REPORT.md](EVAL_REPORT.md), [docs/EVAL_SCOPE_RECONCILIATION.md](EVAL_SCOPE_RECONCILIATION.md), [eval/runner.py](../eval/runner.py), [eval/cases.jsonl](../eval/cases.jsonl), [eval/results/last_run.json](../eval/results/last_run.json) | `pytest tests/test_eval_runner.py tests/test_eval_service.py -q` and `LLM_MODE=demo python -m eval.runner --gate --no-write` | Baseline is deterministic synthetic evidence, not live model quality; Eval Lab's 55-case pass is separate conformance evidence, while its 100-case challenge has no canonical executed run. | +| Eval exists as a repeatable quality signal. | [docs/EVALUATION.md](EVALUATION.md), [docs/EVAL_REPORT.md](EVAL_REPORT.md), [docs/EVAL_SCOPE_RECONCILIATION.md](EVAL_SCOPE_RECONCILIATION.md), [eval/runner.py](../eval/runner.py), [eval/cases.jsonl](../eval/cases.jsonl), [eval/results/last_run.json](../eval/results/last_run.json), [Eval Lab canonical challenge evidence](https://github.com/ashishki/Eval-Ground-Truth-Lab/tree/main/docs/evidence/releases/v0.2.0/gdev-agent-challenge) | `pytest tests/test_eval_runner.py tests/test_eval_service.py -q`, `LLM_MODE=demo python -m eval.runner --gate --no-write`, and verify Eval Lab manifest content address `sha256:656face21f27b496d4d3e8bb0b588824f5737d122c1275c710f3e5b15ff94b4b` | Baseline is deterministic synthetic evidence, not live model quality. Eval Lab's 55-case conformance pass is separate from its canonical 100-case challenge **FAIL** at exact gdev revision `0e4c5f0fd50382bbf12ffd35cfca4632384fb0cc`. | | Runtime exemplar consistency catches obvious triage drift before auto-execution. | [app/exemplar_guard.py](../app/exemplar_guard.py), [eval/exemplars/triage_v1.jsonl](../eval/exemplars/triage_v1.jsonl), [tests/test_agent.py](../tests/test_agent.py), [docs/EVALUATION.md](EVALUATION.md#6-runtime-exemplar-consistency-guard) | `pytest tests/test_agent.py::test_exemplar_consistency_conflict_blocks_auto_approval tests/test_agent.py::test_exemplar_consistency_can_be_disabled -q` | This is a non-authoritative runtime guard; batch evals remain the quality gate. | | Load behavior has scenario targets, local harness assets, and a bounded deterministic report. | [docs/load-profile.md](load-profile.md), [docs/LOAD_TEST_REPORT.md](LOAD_TEST_REPORT.md), [load_tests/locustfile.py](../load_tests/locustfile.py), [load_tests/check_kpis.py](../load_tests/check_kpis.py), [load_tests/results/local-deterministic-2026-06-12/](../load_tests/results/local-deterministic-2026-06-12/) | `pytest tests/test_load_test_fixtures.py -q`, `.venv/bin/python load_tests/check_kpis.py --help`, and `.venv/bin/python load_tests/check_kpis.py --dry-run` | Current report is local deterministic/synthetic evidence; live Locust capacity measurement remains out of scope until a running stack is measured. | | Failure modes have a stable taxonomy and runbook path. | [docs/FAILURE_MODES.md](FAILURE_MODES.md), [docs/SLO_RUNBOOK.md](SLO_RUNBOOK.md), [docs/ARCHITECTURE.md#113-failure-modes-at-the-boundary](ARCHITECTURE.md#113-failure-modes-at-the-boundary), [docs/load-profile.md](load-profile.md) | `pytest tests/test_approval_flow.py tests/test_approval_service.py tests/test_cost_ledger.py tests/test_middleware.py tests/test_isolation.py -q` and `rg -n "FM_DUPLICATE_WEBHOOK_REPLAY|Redis|Postgres|LLM timeout|approval TTL|budget exceedance|SLO|runbook" docs/FAILURE_MODES.md docs/SLO_RUNBOOK.md docs/EVIDENCE_INDEX.md` | Taxonomy and local scenario tests exist; external incident evidence remains out of scope. | diff --git a/docs/STACK_OVERVIEW.md b/docs/STACK_OVERVIEW.md index ecddaed..7b4bb4b 100644 --- a/docs/STACK_OVERVIEW.md +++ b/docs/STACK_OVERVIEW.md @@ -8,7 +8,7 @@ reliable AI/agent systems. | Layer | Repository | Role | Current evidence | | --- | --- | --- | --- | | Governed workflow | `gdev-agent` | Multi-tenant support-triage workflow with webhook intake, guardrails, approval, audit, cost, and observability controls. | 310-test local baseline (2026-07-13), default-Compose FORCE RLS/demo proof, 180-case internal smoke eval, load and isolation evidence. | -| Quality layer | `Eval-Ground-Truth-Lab` | Deterministic regression evaluation framework for structured output, routing, unsafe auto-approval, cost, latency, and adapter behavior. | 55-case live local gdev-agent conformance baseline; separate 100-case challenge dataset/manifest is diagnostic and has no canonical executed run yet. | +| Quality layer | `Eval-Ground-Truth-Lab` | Deterministic regression evaluation framework for structured output, routing, unsafe auto-approval, cost, latency, and adapter behavior. | 55-case live local gdev-agent conformance baseline; [canonical 100-case challenge](https://github.com/ashishki/Eval-Ground-Truth-Lab/tree/main/docs/evidence/releases/v0.2.0/gdev-agent-challenge) against exact revision `0e4c5f0fd50382bbf12ffd35cfca4632384fb0cc` failed its stricter quality gate. | | Runtime layer | `Agent-Runtime-Grid` | Queue-backed runtime for running many AI/agent jobs with retries, timeouts, idempotent finalization, artifacts, metrics, and cost controls. | 100-job smoke, 500-job reliability proof, failure-injection reports, cross-project artifact proof, and 20-job operator-run live-local HTTP proof snapshot. | ## How They Connect diff --git a/docs/evidence/GDEV_P0_TRUTH_REPAIR_2026-07-13.md b/docs/evidence/GDEV_P0_TRUTH_REPAIR_2026-07-13.md index 53782dc..0c47cc3 100644 --- a/docs/evidence/GDEV_P0_TRUTH_REPAIR_2026-07-13.md +++ b/docs/evidence/GDEV_P0_TRUTH_REPAIR_2026-07-13.md @@ -128,9 +128,11 @@ Eval scopes are deliberately separate: - internal gdev-agent smoke: 180 cases, executed above; - Eval Lab conformance baseline: 55 cases, separately committed as 55/55; -- Eval Lab challenge diagnostic: an executable 100-case gate with 90 external - candidate calls and 10 labeled deterministic fault injections, but no - canonical external-system run for this fixed gdev revision yet. +- Eval Lab challenge diagnostic: an executable 100-case gate with 90 local HTTP + candidate calls and 10 labeled deterministic fault injections. At the time + of this repair proof, no canonical run had been recorded. The later + [canonical package](https://github.com/ashishki/Eval-Ground-Truth-Lab/tree/main/docs/evidence/releases/v0.2.0/gdev-agent-challenge) + fixes this gdev revision and records a challenge gate **FAIL**. ### Clean default-Compose isolation proof @@ -164,6 +166,7 @@ all temporary project containers and its network were removed with Compose - The demo tenants, messages, and credentials are synthetic fixtures. - The test and latency numbers are a dated local run, not performance or reliability claims for external traffic. -- The Eval Lab 100-case challenge remains without a canonical external-system - result until its executable gate records this fixed gdev revision; unit-test - fixtures are not promoted as evidence of the service. +- This artifact predates the Eval Lab 100-case canonical result. The later + content-addressed package records this fixed revision, real local HTTP calls, + deterministic fault injections, and the failed gate; unit-test fixtures are + still not promoted as evidence of the service. diff --git a/tests/test_harness_docs.py b/tests/test_harness_docs.py index adf5b0e..4ccec9c 100644 --- a/tests/test_harness_docs.py +++ b/tests/test_harness_docs.py @@ -9,12 +9,17 @@ def test_harness_docs_are_linked_and_define_boundaries() -> None: harness = (ROOT / "docs/HARNESS_CARD.md").read_text(encoding="utf-8") trace = (ROOT / "docs/TRACE_SCHEMA.md").read_text(encoding="utf-8") agents = (ROOT / "AGENTS.md").read_text(encoding="utf-8") + eval_scope = (ROOT / "docs/EVAL_SCOPE_RECONCILIATION.md").read_text(encoding="utf-8") assert "docs/HARNESS_CARD.md" in readme + assert "img.shields.io" not in readme assert "model + prompt/tool loop + guards + approvals + trace + eval" in harness assert "No autonomous long-term memory" in harness assert "gdev-agent-trace-v1" in trace assert "No Silent Workaround Policy" in agents + assert "0e4c5f0fd50382bbf12ffd35cfca4632384fb0cc" in eval_scope + assert "Gate **FAIL**" in eval_scope + assert "sha256:656face21f27b496d4d3e8bb0b588824f5737d122c1275c710f3e5b15ff94b4b" in eval_scope def test_harness_regression_fixture_is_synthetic_and_trace_oriented() -> None: From 65353f8dfd9a2a1d5d1a17c525136732f04596ed Mon Sep 17 00:00:00 2001 From: Artem Shishkin Date: Mon, 13 Jul 2026 17:58:30 +0400 Subject: [PATCH 2/2] docs: define workload portfolio boundaries --- README.md | 65 +++++++++++++++++++++++++++----------- tests/test_harness_docs.py | 5 +++ 2 files changed, 51 insertions(+), 19 deletions(-) diff --git a/README.md b/README.md index 4145e87..7c7f2a3 100644 --- a/README.md +++ b/README.md @@ -1,15 +1,39 @@ # gdev-agent -`gdev-agent` is a governed, multi-tenant LLM workflow reliability system for -game-studio support: it receives support webhooks, blocks unsafe input before -any model call, classifies and extracts structured data with an LLM, routes -risky actions into human approval, and records the resulting audit, cost, and -analytics trail behind one HTTP API. - -Status: local evidence baseline complete. The current stack is pilot-grade: -Docker Compose setup, synthetic demo and eval paths, and repository tests. This -README does not claim production SaaS readiness, external deployment, or live -customer usage. +`gdev-agent` is a governed local support-triage reference workload for +game-studio scenarios. It receives signed support webhooks, blocks unsafe input +before any model call, classifies and extracts structured data, routes risky +actions into human approval, and records an audit/cost trail behind one HTTP +API. + +## Current Maturity + +The supported proof boundary is a tested local Docker Compose workload with +synthetic demo/eval data and database-enforced tenant isolation. Its 55-case +conformance set passes, while the harder published 100-case Eval Lab gate fails. +There is no claimed pilot, external deployment, live customer traffic, +production readiness, or production SLO. + +## Relationship to the Portfolio + +- This repository owns application behavior, signed ingress, approval/audit + flows, tenant controls, and candidate fixes. +- [Eval Ground Truth Lab](https://github.com/ashishki/Eval-Ground-Truth-Lab) + owns external workflow-quality gates. Its canonical 100-case result for this + revision is FAIL and remains authoritative for those challenge outcomes. +- [Agent Runtime Grid](https://github.com/ashishki/Agent-Runtime-Grid) is an + optional execution layer. It does not replace gdev lifecycle/security tests + or Eval Lab quality decisions. +- [AI Workflow Playbook](https://github.com/ashishki/AI_workflow_playbook) is an + independent governance companion, not a runtime dependency. +- The thin umbrella pins compatible revisions and runs integration evidence; it + does not absorb component code or history. + +## Product Boundary and Non-Goals + +The repository demonstrates one bounded local support workflow. It is not a +hosted support platform, customer-data product, generic agent framework, +production multi-tenant SaaS, or proof that the demo classifier generalizes. ## Evidence Path @@ -31,11 +55,15 @@ For a claim-by-claim proof map, start with | Tests | [Current State](#current-state) | 2026-07-13 local baseline: 310 passing tests; rerun locally before relying on it | | Failure modes and SLO/runbook | [docs/FAILURE_MODES.md](docs/FAILURE_MODES.md), [docs/SLO_RUNBOOK.md](docs/SLO_RUNBOOK.md), [docs/observability.md#alert-runbooks](docs/observability.md#alert-runbooks) | Local taxonomy and runbook evidence; external incident evidence is out of scope | | Deployment readiness boundaries | [docs/DEPLOYMENT_READINESS.md](docs/DEPLOYMENT_READINESS.md), [Known Limits](#known-limits) | Secrets checklist, backup/restore notes, local production-like config, and known limitations without production readiness claims | -| Known limits and production changes | [Known Limits](#known-limits), [docs/DEPLOYMENT_READINESS.md](docs/DEPLOYMENT_READINESS.md) | Explicitly bounded as pilot/local evidence, not production SaaS readiness | +| Known limits and production changes | [Known Limits](#known-limits), [docs/DEPLOYMENT_READINESS.md](docs/DEPLOYMENT_READINESS.md) | Explicitly bounded as local reference evidence, not production SaaS readiness | ## Why This Project Exists -Game studios deal with billing disputes, account-access incidents, bug reports, moderation signals, and repetitive gameplay questions at a volume where manual triage becomes slow and brittle. `gdev-agent` is the orchestration layer between inbound support traffic and downstream systems: it keeps routine requests moving, forces human review when confidence or risk is low, and preserves tenant isolation, observability, and cost controls. +The synthetic reference scenario covers billing disputes, account-access +incidents, bug reports, moderation signals, and repetitive gameplay questions. +`gdev-agent` exercises the orchestration boundary between inbound support +traffic and downstream tools: routine paths move through deterministic controls, +while low-confidence or risky actions require review. ## Architecture @@ -77,7 +105,7 @@ The current stack includes FastAPI, Redis, PostgreSQL with Row-Level Security, p This path is aligned to [docker-compose.yml](docker-compose.yml) and is the fastest way to get a healthy local stack. ```bash -git clone https://github.com/your-handle/gdev-agent.git +git clone https://github.com/ashishki/gdev-agent.git cd gdev-agent cp .env.example .env docker compose up --build @@ -235,7 +263,7 @@ Most endpoints outside `/health`, `/webhook`, and `/metrics` require JWT auth pl ## Known Limits -- The project is pilot-grade/local evidence. It has no claimed external +- The project is a tested local reference workload. It has no claimed external deployment, production SaaS readiness, live tenant traffic, or real customer operations. - Demo, eval, and load evidence is synthetic unless a later report explicitly @@ -251,21 +279,20 @@ Most endpoints outside `/health`, `/webhook`, and `/metrics` require JWT auth pl a threshold pass. See [docs/EVAL_SCOPE_RECONCILIATION.md](docs/EVAL_SCOPE_RECONCILIATION.md). - Live load measurements remain out of scope for the current local evidence. - Deployment readiness notes are local/pilot-only and explicitly do not prove + Deployment readiness notes are local-only and explicitly do not prove production readiness. - Live LLM behavior requires a real Anthropic API key and budget controls; the local stack is the supported review path today. ## Current State -The local stack is pilot-grade and feature-complete enough to demonstrate the -governed request pipeline. It includes the multi-tenant storage foundation, +The local stack demonstrates the governed request pipeline. It includes the +tenant-scoped storage foundation, JWT/RBAC boundary, approval hardening, eval APIs with budget enforcement, auth service flows, embedding persistence, RCA clustering with persisted cluster membership, service-layer separation for the main write/auth/eval workflows with read-route extraction still tracked as architecture drift, Dockerized -observability, admin CLI, and the n8n workflow artifacts needed for demo or -pilot-style setups. +observability, admin CLI, and n8n workflow artifacts for local demonstrations. At revision `0e4c5f0fd50382bbf12ffd35cfca4632384fb0cc`, the dated 2026-07-13 local baseline is **310 tests passed** (unit + integration, diff --git a/tests/test_harness_docs.py b/tests/test_harness_docs.py index 4ccec9c..107aae3 100644 --- a/tests/test_harness_docs.py +++ b/tests/test_harness_docs.py @@ -13,6 +13,11 @@ def test_harness_docs_are_linked_and_define_boundaries() -> None: assert "docs/HARNESS_CARD.md" in readme assert "img.shields.io" not in readme + assert "## Current Maturity" in readme + assert "## Relationship to the Portfolio" in readme + assert "## Product Boundary and Non-Goals" in readme + assert "pilot-grade" not in readme + assert "github.com/your-handle" not in readme assert "model + prompt/tool loop + guards + approvals + trace + eval" in harness assert "No autonomous long-term memory" in harness assert "gdev-agent-trace-v1" in trace