diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000..3485863 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,5 @@ +blank_issues_enabled: false +contact_links: + - name: Private security reporting instructions + url: https://github.com/ashishki/gdev-agent/security/policy + about: Use the policy's private email path; do not open a public issue. diff --git a/.github/ISSUE_TEMPLATE/reproducible-bug.yml b/.github/ISSUE_TEMPLATE/reproducible-bug.yml new file mode 100644 index 0000000..ae83ec1 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/reproducible-bug.yml @@ -0,0 +1,96 @@ +name: Reproducible defect +description: Report a deterministic defect in the supported local reference workload. +title: "[bug]: " +body: + - type: markdown + attributes: + value: | + Use this form only for reproducible defects in the supported local boundary. Do not include credentials, private data, or vulnerability details. Security concerns must use the private path in [SECURITY.md](https://github.com/ashishki/gdev-agent/security/policy). + + - type: dropdown + id: surface + attributes: + label: Affected surface + description: Choose the narrowest surface that reproduces the defect. + options: + - Default Docker Compose or tenant/RLS proof + - Signed webhook, guard, routing, or approval flow + - Internal deterministic eval or evidence command + - API, CLI, or local demo + - Documentation or evidence provenance + validations: + required: true + + - type: input + id: revision + attributes: + label: Exact commit SHA + description: Run `git rev-parse HEAD` in the checkout that reproduced the defect. + placeholder: 40-character commit SHA + validations: + required: true + + - type: textarea + id: environment + attributes: + label: Sanitized environment + description: Include OS, Python and Docker versions, run mode, and relevant non-secret configuration. + placeholder: | + OS: + Python: + Docker / Compose: + LLM_MODE: + Relevant configuration (redacted): + validations: + required: true + + - type: textarea + id: prerequisites + attributes: + label: Minimal prerequisites + description: State the clean-checkout setup and synthetic fixtures needed before reproduction. + validations: + required: true + + - type: textarea + id: reproduction + attributes: + label: Reproduction commands + description: Provide the shortest deterministic command sequence, including every required environment variable with values redacted. + render: shell + validations: + required: true + + - type: textarea + id: observed + attributes: + label: Observed result + description: Include the exit code, failing assertion or sanitized response, and a link to any evidence artifact. + validations: + required: true + + - type: textarea + id: expected + attributes: + label: Expected result + description: Point to the repository contract, test, or documented behavior that differs from the observation. + validations: + required: true + + - type: textarea + id: additional + attributes: + label: Additional sanitized evidence + description: Optional logs, traceback, minimal fixture, or regression-test sketch. Remove tokens, credentials, user content, and tenant data. + + - type: checkboxes + id: boundary + attributes: + label: Boundary confirmation + options: + - label: I reproduced this at the exact commit above with synthetic or sanitized data. + required: true + - label: This is a defect report, not a feature or hosted-product roadmap request. + required: true + - label: This report contains no vulnerability details, credentials, private data, or secrets. + required: true diff --git a/README.md b/README.md index 7c7f2a3..0660423 100644 --- a/README.md +++ b/README.md @@ -46,17 +46,25 @@ For a claim-by-claim proof map, start with | Three-project stack map | [docs/STACK_OVERVIEW.md](docs/STACK_OVERVIEW.md) | Explains how gdev-agent, Eval Ground Truth Lab, and Agent Runtime Grid fit together as workflow, quality, and runtime layers | | One-page engineering story | [docs/CASE_STUDY.md](docs/CASE_STUDY.md) | Evidence-backed case study for problem, architecture, controls, eval, load, trade-offs, and production changes | | Architecture and workflow boundaries | [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md), [docs/architecture-diagram.md](docs/architecture-diagram.md) | Implemented local stack with documented gaps and ADRs | +| Human approval decision path | [docs/APPROVAL_FLOW.md](docs/APPROVAL_FLOW.md) | Code-and-test-backed risk, tenant, TTL, reject, and one-time execution flow; not an operator UI or routing-quality claim | | Agent harness boundary | [docs/HARNESS_CARD.md](docs/HARNESS_CARD.md), [docs/TRACE_SCHEMA.md](docs/TRACE_SCHEMA.md), [AGENTS.md](AGENTS.md) | Model + prompt/tool loop + guards + approvals + trace + eval are reviewed as one bounded harness | | Repeatable demo path | [docs/DEMO.md](docs/DEMO.md) | Local Compose demo with deterministic/free mode | | Evaluation discipline | [docs/EVALUATION.md](docs/EVALUATION.md), [docs/EVAL_REPORT.md](docs/EVAL_REPORT.md), [docs/EVAL_SCOPE_RECONCILIATION.md](docs/EVAL_SCOPE_RECONCILIATION.md) | 180-case internal smoke eval, 55-case Eval Lab conformance baseline, and a canonical 100-case challenge run whose stricter gate failed | +| Canonical failure preview | [docs/evidence/GDEV_EVAL_FAILURE_PREVIEW_2026-07-13.md](docs/evidence/GDEV_EVAL_FAILURE_PREVIEW_2026-07-13.md) | Rendered from Eval Lab `v0.2.0` content-addressed raw JSON; five failed thresholds remain visible and limitations are explicit | | Observability | [docs/observability.md](docs/observability.md) | Metrics, traces, logs, and alerting design for local evidence | | Load profile | [docs/load-profile.md](docs/load-profile.md), [docs/LOAD_TEST_REPORT.md](docs/LOAD_TEST_REPORT.md) | Local deterministic/synthetic report and scenario targets; not production capacity claims | | Tenant isolation and security | [docs/TENANT_ISOLATION.md](docs/TENANT_ISOLATION.md), [docs/data-map.md#6-tenant-isolation-model](docs/data-map.md#6-tenant-isolation-model), [docs/ARCHITECTURE.md#7-security-model](docs/ARCHITECTURE.md#7-security-model) | RLS, tenant-scoped JWT, webhook signature, secrets, approval, and cost ledger boundaries | -| Tests | [Current State](#current-state) | 2026-07-13 local baseline: 310 passing tests; rerun locally before relying on it | +| Tests | [Current State](#current-state), [maintainer-surface verification](docs/evidence/GDEV_MAINTAINER_SURFACE_2026-07-13.md) | P0 code baseline: 310 passing tests at `0e4c5f0`; 2026-07-13 maintainer-surface verification: 312 passing tests after two documentation contract tests | | Failure modes and SLO/runbook | [docs/FAILURE_MODES.md](docs/FAILURE_MODES.md), [docs/SLO_RUNBOOK.md](docs/SLO_RUNBOOK.md), [docs/observability.md#alert-runbooks](docs/observability.md#alert-runbooks) | Local taxonomy and runbook evidence; external incident evidence is out of scope | | Deployment readiness boundaries | [docs/DEPLOYMENT_READINESS.md](docs/DEPLOYMENT_READINESS.md), [Known Limits](#known-limits) | Secrets checklist, backup/restore notes, local production-like config, and known limitations without production readiness claims | | Known limits and production changes | [Known Limits](#known-limits), [docs/DEPLOYMENT_READINESS.md](docs/DEPLOYMENT_READINESS.md) | Explicitly bounded as local reference evidence, not production SaaS readiness | +[![Canonical Eval Lab challenge failure preview](docs/assets/gdev-eval-lab-challenge-fail.svg)](docs/evidence/GDEV_EVAL_FAILURE_PREVIEW_2026-07-13.md) + +The image is a preview of pinned negative evidence, not a current quality-pass +badge. Follow it to the exact Eval Lab tag, raw JSON hashes, rendering command, +and interpretation boundary. + ## Why This Project Exists The synthetic reference scenario covers billing disputes, account-access @@ -248,6 +256,9 @@ Most endpoints outside `/health`, `/webhook`, and `/metrics` require JWT auth pl - [docs/HARNESS_CARD.md](docs/HARNESS_CARD.md): agent harness boundary across model, tools, memory, retries, permissions, HITL, trace, and eval. - [docs/TRACE_SCHEMA.md](docs/TRACE_SCHEMA.md): trace completeness contract for debugging, eval, audit, and approval retrospectives. - [docs/architecture-diagram.md](docs/architecture-diagram.md): GitHub-rendered architecture diagram for the main workflow. +- [docs/APPROVAL_FLOW.md](docs/APPROVAL_FLOW.md): implemented human-approval decision path with focused regression commands and limits. +- [docs/evidence/GDEV_EVAL_FAILURE_PREVIEW_2026-07-13.md](docs/evidence/GDEV_EVAL_FAILURE_PREVIEW_2026-07-13.md): content-addressed preview of the canonical failed Eval Lab challenge. +- [docs/evidence/GDEV_MAINTAINER_SURFACE_2026-07-13.md](docs/evidence/GDEV_MAINTAINER_SURFACE_2026-07-13.md): local receipt for the maintainer, visual, full-suite, eval, and Compose gates. - [docs/CASE_STUDY.md](docs/CASE_STUDY.md): concise evidence-backed engineering case study. - [docs/spec.md](docs/spec.md): product scope, API intent, and behavioral contract. - [docs/N8N.md](docs/N8N.md): n8n integration and approval workflow blueprint. @@ -261,6 +272,17 @@ Most endpoints outside `/health`, `/webhook`, and `/metrics` require JWT auth pl - [n8n/README.md](n8n/README.md): workflow assets committed in this repository. - [AGENTS.md](AGENTS.md): operating rules for coding/ops agents working in this repo. +## Maintainer Paths + +- Suspected vulnerabilities must follow the private path in + [SECURITY.md](SECURITY.md); do not disclose them in a public issue. +- Supported local defects use the + [reproducible bug form](https://github.com/ashishki/gdev-agent/issues/new?template=reproducible-bug.yml) + with an exact commit, sanitized environment, and deterministic commands. +- There is no generic feature or hosted-product roadmap intake. The maintained + boundary is the current local reference workload described in + [Current Maturity](#current-maturity). + ## Known Limits - The project is a tested local reference workload. It has no claimed external @@ -303,4 +325,10 @@ demo eval gate and a clean Compose auth/approval demo. Exact commands and bounded outputs are recorded in [docs/evidence/GDEV_P0_TRUTH_REPAIR_2026-07-13.md](docs/evidence/GDEV_P0_TRUTH_REPAIR_2026-07-13.md). +The later maintainer/evidence-surface tranche added two focused documentation +contracts and reran the complete local suite at 312 passing tests, along with +Ruff, the non-writing 180-case eval gate, and an isolated default-Compose RLS +and approval demo. Its bounded receipt is +[docs/evidence/GDEV_MAINTAINER_SURFACE_2026-07-13.md](docs/evidence/GDEV_MAINTAINER_SURFACE_2026-07-13.md). + The main value is the governed request pipeline: webhook in → guardrails → LLM-assisted triage → human approval where needed → auditable execution throughout, with tenant isolation enforced at the database layer and observable at every step. diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..6dacc91 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,54 @@ +# Security Policy + +`gdev-agent` is a maintained local reference workload, not a hosted service. +Security reports are welcome when they affect the supported boundary below. + +## Supported Boundary + +| Surface | Support status | +| --- | --- | +| Current `master` at an identified commit SHA | Supported | +| Documented default local Docker Compose topology | Supported | +| Signed webhook, JWT/RBAC, approval, tenant/RLS, secrets, and output-guard paths | Supported | +| Older commits, unmerged branches, and forks | Not supported | +| Operator-modified deployments and third-party infrastructure | Not supported | +| Live-provider, customer-data, or production operations | No supported deployment exists | + +There is no tagged stable product release or production security SLA. A report +should name the exact commit and demonstrate impact with synthetic or sanitized +data whenever possible. + +Security-relevant examples include cross-tenant access, bypass of JWT/RBAC or +webhook-signature checks, execution without required approval, disclosure of +stored secrets, an output-guard bypass, and unsafe default Compose role or RLS +configuration. Ordinary reproducible defects belong in the +[bug form](https://github.com/ashishki/gdev-agent/issues/new?template=reproducible-bug.yml). + +## Report Privately + +Do **not** open a public issue for a suspected vulnerability. + +1. Email `verter25@gmail.com` with subject `gdev-agent security report`. + Include the affected commit, prerequisites, a minimal reproduction, impact, + and any suggested mitigation. Keep the first message minimal and do not + attach secrets, tokens, customer data, or an exploit against a system you do + not own. A safer detail-transfer channel can be agreed before sending + sensitive material. +2. GitHub private vulnerability reporting is not enabled for this repository + as of 2026-07-13. If the repository's + [Security page](https://github.com/ashishki/gdev-agent/security) later shows + a **Report a vulnerability** button, that enabled private form is also an + acceptable path. Do not rely on the direct advisory URL unless GitHub shows + the feature as enabled. + +Please allow coordinated remediation before public disclosure. This +maintainer-run reference project cannot promise a response or fix deadline, but +reports will be triaged against the explicit support boundary above. Never test +against infrastructure or data without authorization. + +## Public Disclosure + +After a fix is available, the maintainer may publish a GitHub Security Advisory +with affected commits, impact, remediation, and credit if requested. Public +write-ups must not expose credentials, private data, or instructions that would +put an unpatched deployment at avoidable risk. diff --git a/docs/APPROVAL_FLOW.md b/docs/APPROVAL_FLOW.md new file mode 100644 index 0000000..f411fc7 --- /dev/null +++ b/docs/APPROVAL_FLOW.md @@ -0,0 +1,74 @@ +# Human Approval Flow + +This diagram maps the implemented local decision path from a signed webhook to +one-time human approval. It is a code-and-test-backed control-flow view, not an +operator UI, production deployment claim, or proof that every policy decision +is correct. + +```mermaid +flowchart TD + webhook[POST /webhook\nsynthetic or operator request] --> ingress[Per-tenant HMAC + input guard] + ingress --> propose[Classify, extract, propose action\nand apply output/exemplar guards] + propose --> route{Risky action, approval-required tool,\nor bulk/destructive side effect?} + + route -->|No| execute[Execute registered tool] + route -->|Yes| pending[Create tenant-scoped PendingDecision\nRedis TTL + durable pending row] + pending --> response[Return pending_id + draft\nwithout executing the tool] + response --> review[POST /approve\nJWT support_agent or tenant_admin\n+ X-Approve-Secret when configured] + + review --> tenant{JWT tenant owns\npending_id?} + tenant -->|No / missing / expired| deny[401 / 403 / 404\nno action execution] + tenant -->|Yes| consume[Atomic GETDEL\nconsume one time] + consume --> decision{Reviewer decision} + decision -->|Reject| rejected[Record rejection\nno action execution] + decision -->|Approve| approved[Execute original action\nfor original user] + + execute --> audit[Persist audit / action evidence] + rejected --> audit + approved --> audit +``` + +## Implementation Map + +| Diagram boundary | Authoritative implementation | Focused proof | +| --- | --- | --- | +| Risk and tool-side-effect gate | `AgentService.needs_approval()` in `app/agent.py` | `tests/test_agent.py`, `tests/test_tool_registry.py` | +| Tenant-scoped pending state and TTL | `RedisApprovalStore` in `app/approval_store.py` | `tests/test_redis_approval_store.py` | +| JWT role and tenant context | `JWTMiddleware` and `require_role()` | `tests/test_auth.py`, `tests/test_rbac.py` | +| Optional approval secret | `ApprovalService.verify_hmac()` | `tests/test_approval_service.py::test_handle_rejects_wrong_approve_secret` | +| Cross-tenant, expired, reject, and one-time behavior | `AgentService.approve()` | `tests/test_approval_flow.py` | +| Approval/rejection audit records | `_record_approval_event()` and event store calls | `tests/test_approval_flow.py`, `tests/test_approval_service.py` | + +Run the control-flow regression proof from the repository root: + +```bash +PYTHONDONTWRITEBYTECODE=1 LLM_MODE=demo \ + python -m pytest \ + tests/test_approval_flow.py \ + tests/test_approval_service.py \ + tests/test_redis_approval_store.py \ + tests/test_rbac.py -q +``` + +The default local Compose proof additionally checks database role flags, forced +RLS, tenant-A reads, and cross-tenant write rejection: + +```bash +bash scripts/verify_compose_rls.sh +``` + +## Boundaries and Known Limits + +- The diagram shows the repository's current local path. n8n/Telegram button + rendering and external identity-provider controls are outside this proof. +- `X-Approve-Secret` is a defense-in-depth check only when `APPROVE_SECRET` is + configured. JWT role and tenant ownership remain required for `/approve`. +- A wrong tenant receives no pending object because Redis keys are tenant + scoped. Expired and already-consumed decisions also return not found. +- Redis coordinates one-time consumption; Postgres stores durable pending and + approval records in the configured Compose path. This is not a proof of + failover behavior for an externally operated deployment. +- The canonical Eval Lab challenge still reports failed human-routing quality + thresholds. See the [failure preview](evidence/GDEV_EVAL_FAILURE_PREVIEW_2026-07-13.md); + implemented approval mechanics do not imply that candidate policy routes + every risky input correctly. diff --git a/docs/EVIDENCE_INDEX.md b/docs/EVIDENCE_INDEX.md index af3947f..88ec114 100644 --- a/docs/EVIDENCE_INDEX.md +++ b/docs/EVIDENCE_INDEX.md @@ -12,12 +12,15 @@ production SaaS readiness. | problem | [docs/CASE_STUDY.md#problem](CASE_STUDY.md#problem) | Game-studio support triage problem, bounded to local/pilot proof. | | stack role | [docs/STACK_OVERVIEW.md](STACK_OVERVIEW.md) | gdev-agent is the governed workflow layer in a local three-project AI reliability stack. | | architecture | [docs/architecture-diagram.md](architecture-diagram.md), [docs/ARCHITECTURE.md](ARCHITECTURE.md) | Implemented local stack and request flow; no external deployment claim. | +| approval flow | [docs/APPROVAL_FLOW.md](APPROVAL_FLOW.md) | Code/test map for risk gating, tenant ownership, TTL, rejection, and one-time approval; not a routing-quality or operator-UI claim. | | harness boundary | [docs/HARNESS_CARD.md](HARNESS_CARD.md), [docs/TRACE_SCHEMA.md](TRACE_SCHEMA.md), [../AGENTS.md](../AGENTS.md) | Model, prompt/tool loop, tools, memory, retries, permissions, human handoff, trace, and eval are reviewed as one bounded harness. | | control boundaries | [docs/CASE_STUDY.md#control-boundaries](CASE_STUDY.md#control-boundaries), [docs/TENANT_ISOLATION.md](TENANT_ISOLATION.md) | RLS, JWT/RBAC, HMAC, approval, output guard, cost, and observability controls. | | quality evaluation | [docs/EVALUATION.md](EVALUATION.md), [docs/EVAL_REPORT.md](EVAL_REPORT.md), [docs/EVAL_SCOPE_RECONCILIATION.md](EVAL_SCOPE_RECONCILIATION.md) | 180-case internal smoke, 55-case Eval Lab conformance baseline, and a canonical 100-case diagnostic challenge whose stricter gate failed. | +| failed-gate preview | [docs/evidence/GDEV_EVAL_FAILURE_PREVIEW_2026-07-13.md](evidence/GDEV_EVAL_FAILURE_PREVIEW_2026-07-13.md) | Deterministic SVG from Eval Lab `v0.2.0` raw JSON and manifest; the content-addressed package remains authoritative. | | failure behavior | [docs/FAILURE_MODES.md](FAILURE_MODES.md), [docs/SLO_RUNBOOK.md](SLO_RUNBOOK.md) | Local taxonomy and runbook targets; no external incident evidence. | -| baseline metrics | [README.md#current-state](../README.md#current-state), [docs/evidence/GDEV_P0_TRUTH_REPAIR_2026-07-13.md](evidence/GDEV_P0_TRUTH_REPAIR_2026-07-13.md), [docs/LOAD_TEST_REPORT.md](LOAD_TEST_REPORT.md), [docs/EVAL_REPORT.md](EVAL_REPORT.md) | 310-test 2026-07-13 local baseline, eval baseline, and local deterministic load fixture. | +| baseline metrics | [README.md#current-state](../README.md#current-state), [docs/evidence/GDEV_P0_TRUTH_REPAIR_2026-07-13.md](evidence/GDEV_P0_TRUTH_REPAIR_2026-07-13.md), [docs/evidence/GDEV_MAINTAINER_SURFACE_2026-07-13.md](evidence/GDEV_MAINTAINER_SURFACE_2026-07-13.md), [docs/LOAD_TEST_REPORT.md](LOAD_TEST_REPORT.md), [docs/EVAL_REPORT.md](EVAL_REPORT.md) | 310-test P0 code baseline, later 312-test maintainer-surface verification, eval baseline, and local deterministic load fixture. | | demo path | [docs/DEMO.md](DEMO.md), [docs/evidence/GDEV_P0_TRUTH_REPAIR_2026-07-13.md](evidence/GDEV_P0_TRUTH_REPAIR_2026-07-13.md) | Deterministic Compose auth/approval demo output is recorded; no committed video/GIF artifact yet. | +| maintainer and security intake | [SECURITY.md](../SECURITY.md), [reproducible bug form](../.github/ISSUE_TEMPLATE/reproducible-bug.yml) | Private vulnerability path and exact-revision defect intake for the supported local boundary; no hosted-product roadmap. | | known limits | [README.md#known-limits](../README.md#known-limits), [docs/DEPLOYMENT_READINESS.md](DEPLOYMENT_READINESS.md) | Explicit pilot/local limits and missing production evidence. | | production changes | [docs/CASE_STUDY.md#what-would-change-for-production](CASE_STUDY.md#what-would-change-for-production), [docs/DEPLOYMENT_READINESS.md](DEPLOYMENT_READINESS.md) | Required production hardening is documented but not claimed as complete. | @@ -26,9 +29,11 @@ production SaaS readiness. | The engineering story can be reviewed quickly without losing evidence links. | [docs/CASE_STUDY.md](CASE_STUDY.md), [README.md#evidence-path](../README.md#evidence-path) | `rg -n "CASE_STUDY|problem|architecture|failure modes|eval results|load results|production" README.md docs/EVIDENCE_INDEX.md docs/CASE_STUDY.md` | Case study is a local evidence narrative; it inherits the same pilot/local limits as the underlying evidence. | | The three projects have one coherent system map. | [docs/STACK_OVERVIEW.md](STACK_OVERVIEW.md), [README.md#evidence-path](../README.md#evidence-path) | `rg -n "AI Systems Reliability Stack|Eval Ground Truth Lab|Agent Runtime Grid|Provider Strategy|20-job operator-run" docs/STACK_OVERVIEW.md README.md` | Stack map is local evidence; Runtime Grid has an operator-run live-local snapshot, but this still does not claim external adoption or production operations. | | The system has a governed, multi-tenant LLM workflow architecture for support intake, triage, approval, and audit. | [docs/ARCHITECTURE.md](ARCHITECTURE.md), [docs/architecture-diagram.md](architecture-diagram.md), [README.md](../README.md) | `rg -n "WebhookService|ApprovalService|OutputGuard|Row-Level Security" docs/ARCHITECTURE.md README.md` | Architecture proof is current repo evidence, not an external deployment review. | +| The human-approval mechanics are reviewable from risk decision through one-time tenant-scoped execution. | [docs/APPROVAL_FLOW.md](APPROVAL_FLOW.md), [app/agent.py](../app/agent.py), [app/approval_store.py](../app/approval_store.py) | `pytest tests/test_approval_flow.py tests/test_approval_service.py tests/test_redis_approval_store.py tests/test_rbac.py -q` | This proves local control-flow behavior, not that the candidate routes every risky input correctly; the canonical challenge's human-routing gates failed. | | The agent is documented as a measurable harness, not just a model call. | [docs/HARNESS_CARD.md](HARNESS_CARD.md), [docs/TRACE_SCHEMA.md](TRACE_SCHEMA.md), [../AGENTS.md](../AGENTS.md), [../eval/harness_regression.jsonl](../eval/harness_regression.jsonl), [../tests/test_harness_docs.py](../tests/test_harness_docs.py) | `pytest tests/test_harness_docs.py -q` and `rg -n "Harness Boundary|Trace Requirements|No Silent Workaround" docs/HARNESS_CARD.md docs/TRACE_SCHEMA.md AGENTS.md` | The fixture is trace-oriented and synthetic; it is not yet wired into the main eval runner adapter. | | The demo path can exercise the local approval workflow. | [docs/DEMO.md](DEMO.md), [scripts/demo.py](../scripts/demo.py), [scripts/demo.sh](../scripts/demo.sh), [docs/evidence/GDEV_P0_TRUTH_REPAIR_2026-07-13.md](evidence/GDEV_P0_TRUTH_REPAIR_2026-07-13.md) | `docker compose exec -T agent python scripts/demo.py --url http://localhost:8000 --llm-mode demo` | Demo evidence is deterministic and local-only; it is not an external deployment proof. | | Eval exists as a repeatable quality signal. | [docs/EVALUATION.md](EVALUATION.md), [docs/EVAL_REPORT.md](EVAL_REPORT.md), [docs/EVAL_SCOPE_RECONCILIATION.md](EVAL_SCOPE_RECONCILIATION.md), [eval/runner.py](../eval/runner.py), [eval/cases.jsonl](../eval/cases.jsonl), [eval/results/last_run.json](../eval/results/last_run.json), [Eval Lab canonical challenge evidence](https://github.com/ashishki/Eval-Ground-Truth-Lab/tree/main/docs/evidence/releases/v0.2.0/gdev-agent-challenge) | `pytest tests/test_eval_runner.py tests/test_eval_service.py -q`, `LLM_MODE=demo python -m eval.runner --gate --no-write`, and verify Eval Lab manifest content address `sha256:656face21f27b496d4d3e8bb0b588824f5737d122c1275c710f3e5b15ff94b4b` | Baseline is deterministic synthetic evidence, not live model quality. Eval Lab's 55-case conformance pass is separate from its canonical 100-case challenge **FAIL** at exact gdev revision `0e4c5f0fd50382bbf12ffd35cfca4632384fb0cc`. | +| The failed challenge is visible without losing machine-readable provenance or limits. | [docs/evidence/GDEV_EVAL_FAILURE_PREVIEW_2026-07-13.md](evidence/GDEV_EVAL_FAILURE_PREVIEW_2026-07-13.md), [generated SVG](assets/gdev-eval-lab-challenge-fail.svg), [renderer](../scripts/render_eval_failure_preview.py), [Eval Lab `v0.2.0` raw summary](https://github.com/ashishki/Eval-Ground-Truth-Lab/blob/v0.2.0/docs/evidence/releases/v0.2.0/gdev-agent-challenge/challenge-run.json) | Run the documented renderer with `--check`; it verifies the raw summary SHA against manifest content address `sha256:656face21f27b496d4d3e8bb0b588824f5737d122c1275c710f3e5b15ff94b4b` before rendering. | The preview is historical local/synthetic evidence for candidate `0e4c5f0`; it is not a rerun of later commits or a production claim. | | Runtime exemplar consistency catches obvious triage drift before auto-execution. | [app/exemplar_guard.py](../app/exemplar_guard.py), [eval/exemplars/triage_v1.jsonl](../eval/exemplars/triage_v1.jsonl), [tests/test_agent.py](../tests/test_agent.py), [docs/EVALUATION.md](EVALUATION.md#6-runtime-exemplar-consistency-guard) | `pytest tests/test_agent.py::test_exemplar_consistency_conflict_blocks_auto_approval tests/test_agent.py::test_exemplar_consistency_can_be_disabled -q` | This is a non-authoritative runtime guard; batch evals remain the quality gate. | | Load behavior has scenario targets, local harness assets, and a bounded deterministic report. | [docs/load-profile.md](load-profile.md), [docs/LOAD_TEST_REPORT.md](LOAD_TEST_REPORT.md), [load_tests/locustfile.py](../load_tests/locustfile.py), [load_tests/check_kpis.py](../load_tests/check_kpis.py), [load_tests/results/local-deterministic-2026-06-12/](../load_tests/results/local-deterministic-2026-06-12/) | `pytest tests/test_load_test_fixtures.py -q`, `.venv/bin/python load_tests/check_kpis.py --help`, and `.venv/bin/python load_tests/check_kpis.py --dry-run` | Current report is local deterministic/synthetic evidence; live Locust capacity measurement remains out of scope until a running stack is measured. | | Failure modes have a stable taxonomy and runbook path. | [docs/FAILURE_MODES.md](FAILURE_MODES.md), [docs/SLO_RUNBOOK.md](SLO_RUNBOOK.md), [docs/ARCHITECTURE.md#113-failure-modes-at-the-boundary](ARCHITECTURE.md#113-failure-modes-at-the-boundary), [docs/load-profile.md](load-profile.md) | `pytest tests/test_approval_flow.py tests/test_approval_service.py tests/test_cost_ledger.py tests/test_middleware.py tests/test_isolation.py -q` and `rg -n "FM_DUPLICATE_WEBHOOK_REPLAY|Redis|Postgres|LLM timeout|approval TTL|budget exceedance|SLO|runbook" docs/FAILURE_MODES.md docs/SLO_RUNBOOK.md docs/EVIDENCE_INDEX.md` | Taxonomy and local scenario tests exist; external incident evidence remains out of scope. | @@ -36,6 +41,7 @@ production SaaS readiness. | SLO/runbook thinking is present without production SLA claims. | [docs/SLO_RUNBOOK.md](SLO_RUNBOOK.md), [docs/load-profile.md](load-profile.md), [docs/observability.md#alert-runbooks](observability.md#alert-runbooks) | `rg -n "p50|p95|p99|runbook|alert|SLO|production SLA" docs/SLO_RUNBOOK.md docs/load-profile.md docs/observability.md` | Local targets are documented; measured external production SLOs are out of scope. | | Observability covers metrics, traces, logs, dashboards, and tenant-safe labels. | [docs/observability.md](observability.md), [docker/grafana/provisioning/dashboards/gdev-agent.json](../docker/grafana/provisioning/dashboards/gdev-agent.json), [tests/test_observability.py](../tests/test_observability.py), [tests/test_metrics.py](../tests/test_metrics.py) | `pytest tests/test_observability.py tests/test_metrics.py -q` and `rg -n "gdev_requests_total|tenant_hash|trace|dashboard|observability" docs/observability.md docker/grafana/provisioning/dashboards/gdev-agent.json` | Current proof is local dashboard JSON and tests; external managed dashboards remain out of scope. | | CI runs the repository safety net. | [.github/workflows/ci.yml](../.github/workflows/ci.yml) | `rg -n "pytest|ruff|eval|compose-isolation|verify_compose_rls" .github/workflows/ci.yml` | CI exercises source gates plus default-Compose RLS and demo topology; external deployment checks remain out of scope. | -| Tests back the current implementation baseline. | [README.md#current-state](../README.md#current-state), [tests/](../tests), [docs/evidence/GDEV_P0_TRUTH_REPAIR_2026-07-13.md](evidence/GDEV_P0_TRUTH_REPAIR_2026-07-13.md) | `PYTHONDONTWRITEBYTECODE=1 LLM_MODE=demo python -m pytest tests/ -q --tb=short` | The dated baseline is local evidence; rerun before using it as current proof. | +| Tests back the current implementation baseline. | [README.md#current-state](../README.md#current-state), [tests/](../tests), [docs/evidence/GDEV_P0_TRUTH_REPAIR_2026-07-13.md](evidence/GDEV_P0_TRUTH_REPAIR_2026-07-13.md), [docs/evidence/GDEV_MAINTAINER_SURFACE_2026-07-13.md](evidence/GDEV_MAINTAINER_SURFACE_2026-07-13.md) | `PYTHONDONTWRITEBYTECODE=1 LLM_MODE=demo python -m pytest tests/ -q --tb=short` | Both dated runs are local evidence; exact-head GitHub Actions remains a separate merge gate. | | Deployment readiness knowledge is documented without production readiness claims. | [docs/DEPLOYMENT_READINESS.md](DEPLOYMENT_READINESS.md), [.env.example](../.env.example), [README.md#known-limits](../README.md#known-limits) | `rg -n "DEPLOYMENT_READINESS|secrets checklist|backup|restore|known limitations|production readiness" README.md docs/EVIDENCE_INDEX.md docs/DEPLOYMENT_READINESS.md .env.example` | Current proof is local/pilot setup knowledge only; cloud controls, managed backups, restore drills, and external deployment remain out of scope. | +| Security and defect intake are bounded to maintained, reproducible surfaces. | [SECURITY.md](../SECURITY.md), [reproducible bug form](../.github/ISSUE_TEMPLATE/reproducible-bug.yml), [issue configuration](../.github/ISSUE_TEMPLATE/config.yml) | `pytest tests/test_harness_docs.py -q` | Private vulnerability reporting availability depends on GitHub/account access; no response SLA or broad feature roadmap is claimed. | | Known limits and production changes are explicit. | [README.md#known-limits](../README.md#known-limits), [docs/DEPLOYMENT_READINESS.md](DEPLOYMENT_READINESS.md), [docs/CASE_STUDY.md#what-would-change-for-production](CASE_STUDY.md#what-would-change-for-production) | `rg -n "known limits|production SaaS|pilot|deployment readiness" README.md docs/DEPLOYMENT_READINESS.md docs/CASE_STUDY.md` | Production hardening remains future work; the public docs state those limits instead of claiming readiness. | diff --git a/docs/assets/gdev-eval-lab-challenge-fail.svg b/docs/assets/gdev-eval-lab-challenge-fail.svg new file mode 100644 index 0000000..525be8c --- /dev/null +++ b/docs/assets/gdev-eval-lab-challenge-fail.svg @@ -0,0 +1,68 @@ + + + Canonical Eval Lab gdev-agent challenge: failed gate + The v0.2.0 local synthetic challenge recorded a failed gate for gdev-agent revision 0e4c5f0fd50382bbf12ffd35cfca4632384fb0cc. Five cards show every failed threshold from the content-addressed raw evidence. This is not production proof. + {"component_image_digest":"sha256:7dc9fef2ec6fe25745405546ec69f6a6f64c1bfa9f052dc54abfd65498a6f6da","component_revision":"0e4c5f0fd50382bbf12ffd35cfca4632384fb0cc","content_address":"sha256:656face21f27b496d4d3e8bb0b588824f5737d122c1275c710f3e5b15ff94b4b","dataset_hash":"151e5eec83373b92cf263aa1f32edb26ed780c260ce32a9d084ba8f3f38e53b0","gate_passed":false,"limitations":"local synthetic evidence; not production quality, usage, reliability, or SLO proof","run_id":"gdev-challenge-0e4c5f0-canonical-20260713-v3","source_challenge_run_sha256":"d4bb0dec70d75de8d33d16f583a01ecbc17bf202ae04509d3dec63087a7b3a3b"} + + + + CONTENT-ADDRESSED NEGATIVE EVIDENCE + Eval Lab challenge · gdev-agent + Eval Lab v0.2.0 · 100 synthetic cases · candidate 0e4c5f0fd503 + + + FAIL + 5 of 12 threshold checks failed + + + + Blocking failures + 58 + maximum 0 + + + + Classification + 24.44% + minimum 70% + + + + Human review required + 46 + minimum 80 + + + + Escalation recall + 46% + minimum 95% + + + + Unexpected failures + 68 + maximum 20 + + + Execution: 90 candidate HTTP calls + 10 labeled deterministic provider-fault injections. + Diagnostic aggregate: 32% reconciled pass. Other checks passed invalid-output, cost, local-latency, and unsafe-auto-approval bounds. + + run gdev-challenge-0e4c5f0-canonical-20260713-v3 + manifest sha256:656face21f27b496d4d3e8bb0b588824f5737d122c1275c710f3e5b15ff94b4b + challenge-run.json sha256:d4bb0dec70d75de8d33d16f583a01ecbc17bf202ae04509d3dec63087a7b3a3b + Local synthetic diagnostic only · no production quality, traffic, adoption, reliability, or SLO claim + diff --git a/docs/evidence/GDEV_EVAL_FAILURE_PREVIEW_2026-07-13.md b/docs/evidence/GDEV_EVAL_FAILURE_PREVIEW_2026-07-13.md new file mode 100644 index 0000000..2d37f3d --- /dev/null +++ b/docs/evidence/GDEV_EVAL_FAILURE_PREVIEW_2026-07-13.md @@ -0,0 +1,78 @@ +# Canonical Eval Lab Failure Preview + +[![Canonical Eval Lab gdev-agent challenge failure](../assets/gdev-eval-lab-challenge-fail.svg)](https://github.com/ashishki/Eval-Ground-Truth-Lab/blob/v0.2.0/docs/evidence/releases/v0.2.0/gdev-agent-challenge/challenge-report.md) + +This preview is a deterministic rendering of Eval Ground Truth Lab's canonical +`v0.2.0` gdev-agent challenge package. It makes the negative result scannable +without replacing the raw report or machine-readable artifacts. + +## Exact Provenance + +| Field | Pinned value | +| --- | --- | +| Eval Lab release | Annotated tag `v0.2.0`, peeled commit `64e6ab384883604e210ba7420529aaecf37f30ed` | +| Evidence content address | `sha256:656face21f27b496d4d3e8bb0b588824f5737d122c1275c710f3e5b15ff94b4b` | +| Canonical run | `gdev-challenge-0e4c5f0-canonical-20260713-v3` | +| gdev-agent candidate | `0e4c5f0fd50382bbf12ffd35cfca4632384fb0cc`, clean worktree | +| Candidate image | `sha256:7dc9fef2ec6fe25745405546ec69f6a6f64c1bfa9f052dc54abfd65498a6f6da` | +| Dataset | `challenge_v1`, 100 synthetic cases, logical hash `151e5eec83373b92cf263aa1f32edb26ed780c260ce32a9d084ba8f3f38e53b0` | +| Raw summary | [`challenge-run.json`](https://github.com/ashishki/Eval-Ground-Truth-Lab/blob/v0.2.0/docs/evidence/releases/v0.2.0/gdev-agent-challenge/challenge-run.json), SHA-256 `d4bb0dec70d75de8d33d16f583a01ecbc17bf202ae04509d3dec63087a7b3a3b` | +| Manifest | [`sha256-656face...manifest.json`](https://github.com/ashishki/Eval-Ground-Truth-Lab/blob/v0.2.0/docs/evidence/releases/v0.2.0/gdev-agent-challenge/sha256-656face21f27b496d4d3e8bb0b588824f5737d122c1275c710f3e5b15ff94b4b.manifest.json), file SHA-256 `b793cc7cc63e0ded101ce14ea879fa61378d752ebf7426b8da8d50dd00bddd1d` | +| Generated SVG | [`docs/assets/gdev-eval-lab-challenge-fail.svg`](../assets/gdev-eval-lab-challenge-fail.svg), SHA-256 `e422da461ba3c133667d65dce3068cf65b689d9f98848c5c8184e9b89e051110` | + +The renderer first checks the pinned manifest file digest and content address; +the manifest then binds the raw summary digest before any metric is read. It +also rejects a passing gate, an unrecognized schema, a digest mismatch, or a +component revision other than the canonical candidate. + +## Recorded Result + +The challenge gate is **FAIL**. Five of 12 configured threshold checks failed: + +| Signal | Observed | Gate | +| --- | ---: | ---: | +| Reconciled pass rate | `0.32` | Diagnostic aggregate | +| Blocking failures | `58` | `<= 0` | +| Classification accuracy | `0.244444` | `>= 0.70` | +| Human review required | `46` | `>= 80` | +| Human-escalation recall | `0.46` | `>= 0.95` | +| Unexpected failures | `68` | `<= 20` | + +The run made 90 candidate HTTP calls and reconciled 10 labeled deterministic +provider-fault injections. Expected-failure match for that injected slice was +`1.0`; those injections are harness behavior, not observed provider outages. +Passing invalid-output, cost, local-latency, and aggregate +unsafe-auto-approval thresholds do not override the overall failure. + +## Reproduce the Preview + +Check out Eval Lab at the exact `v0.2.0` tag next to this repository, then run: + +```bash +python scripts/render_eval_failure_preview.py \ + --summary ../Eval-Ground-Truth-Lab/docs/evidence/releases/v0.2.0/gdev-agent-challenge/challenge-run.json \ + --manifest ../Eval-Ground-Truth-Lab/docs/evidence/releases/v0.2.0/gdev-agent-challenge/sha256-656face21f27b496d4d3e8bb0b588824f5737d122c1275c710f3e5b15ff94b4b.manifest.json \ + --output docs/assets/gdev-eval-lab-challenge-fail.svg \ + --check +``` + +Remove `--check` to regenerate the SVG. Verify the tag before relying on the +adjacent checkout: + +```bash +git -C ../Eval-Ground-Truth-Lab rev-parse 'v0.2.0^{}' +``` + +Expected value: `64e6ab384883604e210ba7420529aaecf37f30ed`. + +## Interpretation Boundary + +- This is a historical result for the exact gdev-agent candidate above. Later + documentation or maintainer-surface commits are not silently treated as a + rerun, pass, or behavior change. The preview is not a rerun of current + `master`. +- The dataset and service environment are local/synthetic. This does not prove + production quality, customer behavior, adoption, provider reliability, + tenant isolation in an external deployment, or a production SLO. +- The SVG is a preview. Eval Lab's content-addressed manifest, JSON summary, + full raw run, and report remain authoritative if any presentation differs. diff --git a/docs/evidence/GDEV_MAINTAINER_SURFACE_2026-07-13.md b/docs/evidence/GDEV_MAINTAINER_SURFACE_2026-07-13.md new file mode 100644 index 0000000..05843a0 --- /dev/null +++ b/docs/evidence/GDEV_MAINTAINER_SURFACE_2026-07-13.md @@ -0,0 +1,99 @@ +# gdev-agent Maintainer and Evidence Surface Verification + +Date: 2026-07-13 + +Scope: security/defect intake, human-approval diagram, and canonical Eval Lab +failure preview + +Starting revision: `64b9cec85d990d63ddf17e1ccf26ad5b73114185` + +Verification branch: `agent/maintainer-evidence-surface` + +This is local verification for documentation, a deterministic renderer, issue +configuration, and their contract tests. It does not claim a new gdev-agent +quality result, production readiness, external use, or a security audit. The +canonical challenge preview remains historical evidence for exact candidate +`0e4c5f0fd50382bbf12ffd35cfca4632384fb0cc`. + +## Source and Contract Gates + +Commands used the existing project virtual environment while the working +directory and imported repository source were the isolated clean clone. + +```bash +ruff check app/ tests/ scripts/ eval/ \ + alembic/versions/0007_enforce_compose_rls_topology.py +ruff format --check app/ tests/ scripts/ eval/ \ + alembic/versions/0007_enforce_compose_rls_topology.py +PYTHONDONTWRITEBYTECODE=1 LLM_MODE=demo \ + python -m pytest tests/ -q --tb=short +PYTHONDONTWRITEBYTECODE=1 LLM_MODE=demo \ + python -m eval.runner --gate --no-write +``` + +Observed results: + +```text +ruff check: All checks passed! +ruff format: 98 files already formatted +pytest: 312 passed, 45 warnings in 109.15s +eval: threshold_result.passed=true; dataset SHA-256 + 8db471b52ea78f6bf9daa3993630f57083277660f31ced8e85790860e0b98400 +``` + +The 45 warnings are the existing Alembic `path_separator` deprecation warning +from container-backed fixtures. The eval command used `--no-write`, so its +machine-local timing did not replace the committed dated baseline. + +## Content-Addressed Preview Check + +Eval Lab annotated tag `v0.2.0` peeled to +`64e6ab384883604e210ba7420529aaecf37f30ed`. The renderer read the tagged +`challenge-run.json` and manifest from an adjacent checkout, verified raw +summary SHA-256 +`d4bb0dec70d75de8d33d16f583a01ecbc17bf202ae04509d3dec63087a7b3a3b`, +and accepted manifest content address +`sha256:656face21f27b496d4d3e8bb0b588824f5737d122c1275c710f3e5b15ff94b4b`. + +```bash +python scripts/render_eval_failure_preview.py \ + --summary ../Eval-Ground-Truth-Lab/docs/evidence/releases/v0.2.0/gdev-agent-challenge/challenge-run.json \ + --manifest ../Eval-Ground-Truth-Lab/docs/evidence/releases/v0.2.0/gdev-agent-challenge/sha256-656face21f27b496d4d3e8bb0b588824f5737d122c1275c710f3e5b15ff94b4b.manifest.json \ + --output docs/assets/gdev-eval-lab-challenge-fail.svg \ + --check +``` + +Result: exit `0`. Generated SVG SHA-256: +`e422da461ba3c133667d65dce3068cf65b689d9f98848c5c8184e9b89e051110`. +An intentionally modified temporary copy of `challenge-run.json` was rejected +with `challenge-run.json does not match its manifest digest`; no preview was +accepted from the tampered input. +The SVG was also parsed as XML and rendered locally to a temporary PNG for +visual inspection; the PNG was not committed. + +## Default Compose Proof + +An isolated Compose project named `gdev_maintainer_surface` built and started +`postgres`, `redis`, `migrate`, and `agent`. The repository proof returned: + +```text +PASS role_flags gdev_app rolsuper=f rolbypassrls=f +PASS table_topology expected=16 owner=gdev_owner rls=enabled force_rls=true +PASS tenant_a_isolation rows=1 tenant_ids=1 tenant=aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa +PASS cross_tenant_insert rejected=true persisted_rows=0 +``` + +The deterministic demo then completed health, auth, signed webhook, pending +audit lookup, one-time approval, and metrics checks. All project containers, +volumes, and its network were removed after verification. + +Local verification used the available `docker-compose` 1.29.2 fallback. GitHub +Actions remains the separate exact-head gate for the workflow's `docker compose` +execution. + +## Workspace Boundary + +All changes and checks ran in +`.portfolio-audit-worktrees/gdev-maintainer-surface`. The primary checkout's +pre-existing dirty `app/jobs/rca_clusterer.py` and +`eval/__pycache__/runner.cpython-312.pyc` were not edited, staged, or reset. diff --git a/scripts/render_eval_failure_preview.py b/scripts/render_eval_failure_preview.py new file mode 100644 index 0000000..6cdf8ab --- /dev/null +++ b/scripts/render_eval_failure_preview.py @@ -0,0 +1,227 @@ +"""Render a deterministic SVG preview from Eval Lab's canonical gdev evidence.""" + +from __future__ import annotations + +import argparse +import hashlib +import html +import json +from pathlib import Path +from typing import Any + +CANONICAL_CONTENT_ADDRESS = ( + "sha256:656face21f27b496d4d3e8bb0b588824f5737d122c1275c710f3e5b15ff94b4b" +) +CANONICAL_MANIFEST_SHA256 = "b793cc7cc63e0ded101ce14ea879fa61378d752ebf7426b8da8d50dd00bddd1d" +CANONICAL_SUMMARY_SHA256 = "d4bb0dec70d75de8d33d16f583a01ecbc17bf202ae04509d3dec63087a7b3a3b" +CANONICAL_COMPONENT_REVISION = "0e4c5f0fd50382bbf12ffd35cfca4632384fb0cc" + + +def _load_json(path: Path) -> dict[str, Any]: + value = json.loads(path.read_text(encoding="utf-8")) + if not isinstance(value, dict): + raise ValueError(f"expected a JSON object: {path}") + return value + + +def _sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as source: + for chunk in iter(lambda: source.read(65536), b""): + digest.update(chunk) + return digest.hexdigest() + + +def _artifact_hash(manifest: dict[str, Any], path: str) -> str: + for artifact in manifest.get("artifacts", []): + if artifact.get("path") == path: + value = artifact.get("sha256") + if isinstance(value, str): + return value + raise ValueError(f"manifest does not bind {path}") + + +def _validate(summary_path: Path, manifest_path: Path) -> tuple[dict[str, Any], dict[str, Any]]: + summary = _load_json(summary_path) + manifest = _load_json(manifest_path) + + if summary.get("schema_version") != "gdev-agent-challenge-run-v1": + raise ValueError("unsupported challenge summary schema") + if manifest.get("schema_version") != "eval-lab-evidence-v1": + raise ValueError("unsupported evidence manifest schema") + if _sha256(manifest_path) != CANONICAL_MANIFEST_SHA256: + raise ValueError("manifest does not match the pinned canonical digest") + if manifest.get("content_address") != CANONICAL_CONTENT_ADDRESS: + raise ValueError("manifest does not match the pinned content address") + if summary.get("gate", {}).get("passed") is not False: + raise ValueError("this renderer only accepts a recorded failed gate") + if manifest.get("metadata", {}).get("gate_passed") is not False: + raise ValueError("manifest and failed gate do not agree") + + expected_summary_hash = _artifact_hash(manifest, "challenge-run.json") + if expected_summary_hash != CANONICAL_SUMMARY_SHA256: + raise ValueError("manifest binds an unexpected challenge summary digest") + if _sha256(summary_path) != expected_summary_hash: + raise ValueError("challenge-run.json does not match its manifest digest") + + component_revision = summary.get("provenance", {}).get("component_revision") + manifest_revision = manifest.get("metadata", {}).get("component_revision") + if component_revision != manifest_revision: + raise ValueError("component revision differs between summary and manifest") + if component_revision != CANONICAL_COMPONENT_REVISION: + raise ValueError("component revision is not the pinned canonical candidate") + + required_metrics = { + "blocking_failure_count", + "candidate_scope_case_count", + "classification_accuracy", + "diagnostic_failure_count", + "human_escalation_recall", + "human_review_required_count", + "reconciled_pass_rate", + "total_case_count", + "unexpected_fail_count", + } + if not required_metrics <= summary.get("metrics", {}).keys(): + raise ValueError("challenge summary is missing required metrics") + + failed = summary.get("gate", {}).get("failed_thresholds") + if not isinstance(failed, list) or not failed: + raise ValueError("failed gate must name its failed thresholds") + return summary, manifest + + +def _percent(value: object, digits: int = 0) -> str: + number = float(value) * 100 + return f"{number:.{digits}f}%" + + +def render_svg(summary: dict[str, Any], manifest: dict[str, Any]) -> str: + metrics = summary["metrics"] + thresholds = summary["threshold_results"] + run = summary["run"] + provenance = summary["provenance"] + content_address = str(manifest["content_address"]) + summary_hash = _artifact_hash(manifest, "challenge-run.json") + revision = str(provenance["component_revision"]) + failed_count = len(summary["gate"]["failed_thresholds"]) + threshold_count = len(thresholds) + + cards = [ + ( + "Blocking failures", + str(metrics["blocking_failure_count"]), + "maximum 0", + ), + ( + "Classification", + _percent(metrics["classification_accuracy"], 2), + "minimum 70%", + ), + ( + "Human review required", + str(metrics["human_review_required_count"]), + "minimum 80", + ), + ( + "Escalation recall", + _percent(metrics["human_escalation_recall"]), + "minimum 95%", + ), + ( + "Unexpected failures", + str(metrics["unexpected_fail_count"]), + "maximum 20", + ), + ] + + card_svg: list[str] = [] + for index, (label, value, limit) in enumerate(cards): + x = 54 + index * 220 + card_svg.append( + f""" + + {html.escape(label)} + {html.escape(value)} + {html.escape(limit)} + """ + ) + + metadata = { + "content_address": content_address, + "source_challenge_run_sha256": summary_hash, + "run_id": run["run_id"], + "component_revision": revision, + "component_image_digest": provenance["component_image_digest"], + "dataset_hash": summary["dataset"]["dataset_hash"], + "gate_passed": False, + "limitations": "local synthetic evidence; not production quality, usage, reliability, or SLO proof", + } + metadata_text = html.escape(json.dumps(metadata, sort_keys=True, separators=(",", ":"))) + + return f""" + + Canonical Eval Lab gdev-agent challenge: failed gate + The v0.2.0 local synthetic challenge recorded a failed gate for gdev-agent revision {html.escape(revision)}. Five cards show every failed threshold from the content-addressed raw evidence. This is not production proof. + {metadata_text} + + + + CONTENT-ADDRESSED NEGATIVE EVIDENCE + Eval Lab challenge · gdev-agent + Eval Lab v0.2.0 · {int(metrics["total_case_count"])} synthetic cases · candidate {html.escape(revision[:12])} + + + FAIL + {failed_count} of {threshold_count} threshold checks failed + +{chr(10).join(card_svg)} + + Execution: {int(metrics["candidate_scope_case_count"])} candidate HTTP calls + {int(metrics["diagnostic_failure_count"])} labeled deterministic provider-fault injections. + Diagnostic aggregate: {_percent(metrics["reconciled_pass_rate"])} reconciled pass. Other checks passed invalid-output, cost, local-latency, and unsafe-auto-approval bounds. + + run {html.escape(str(run["run_id"]))} + manifest {html.escape(content_address)} + challenge-run.json sha256:{html.escape(summary_hash)} + Local synthetic diagnostic only · no production quality, traffic, adoption, reliability, or SLO claim + +""" + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--summary", type=Path, required=True, help="canonical challenge-run.json") + parser.add_argument("--manifest", type=Path, required=True, help="adjacent evidence manifest") + parser.add_argument("--output", type=Path, required=True, help="SVG output path") + parser.add_argument( + "--check", action="store_true", help="fail if output differs instead of writing it" + ) + args = parser.parse_args() + + summary, manifest = _validate(args.summary, args.manifest) + rendered = render_svg(summary, manifest) + if args.check: + if not args.output.exists() or args.output.read_text(encoding="utf-8") != rendered: + raise SystemExit(f"preview is stale: {args.output}") + return 0 + + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(rendered, encoding="utf-8") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_harness_docs.py b/tests/test_harness_docs.py index 107aae3..1dad06a 100644 --- a/tests/test_harness_docs.py +++ b/tests/test_harness_docs.py @@ -1,3 +1,4 @@ +import hashlib import json from pathlib import Path @@ -47,3 +48,78 @@ def test_harness_regression_fixture_is_synthetic_and_trace_oriented() -> None: } assert all(case["expected_trace_events"] for case in cases) assert all(case["expected_route"] != "auto_execute" for case in cases) + + +def test_maintainer_intake_is_private_bounded_and_reproducible() -> None: + readme = (ROOT / "README.md").read_text(encoding="utf-8") + security = (ROOT / "SECURITY.md").read_text(encoding="utf-8") + bug_form = (ROOT / ".github/ISSUE_TEMPLATE/reproducible-bug.yml").read_text(encoding="utf-8") + issue_config = (ROOT / ".github/ISSUE_TEMPLATE/config.yml").read_text(encoding="utf-8") + + assert "verter25@gmail.com" in security + assert "GitHub private vulnerability reporting is not enabled" in security + assert "https://github.com/ashishki/gdev-agent/security/policy" in issue_config + assert "security/advisories/new" not in issue_config + assert "Do **not** open a public issue" in security + assert "Current `master` at an identified commit SHA" in security + assert "no tagged stable product release" in security + assert "cannot promise a response or fix deadline" in security + + assert "Exact commit SHA" in bug_form + assert "Reproduction commands" in bug_form + assert "synthetic or sanitized data" in bug_form + assert "not a feature or hosted-product roadmap request" in bug_form + assert "blank_issues_enabled: false" in issue_config + assert not list((ROOT / ".github/ISSUE_TEMPLATE").glob("*feature*")) + + assert "## Maintainer Paths" in readme + assert "SECURITY.md" in readme + assert "reproducible-bug.yml" in readme + + +def test_visual_evidence_surfaces_are_pinned_and_bounded() -> None: + readme = (ROOT / "README.md").read_text(encoding="utf-8") + evidence_index = (ROOT / "docs/EVIDENCE_INDEX.md").read_text(encoding="utf-8") + approval = (ROOT / "docs/APPROVAL_FLOW.md").read_text(encoding="utf-8") + preview = (ROOT / "docs/evidence/GDEV_EVAL_FAILURE_PREVIEW_2026-07-13.md").read_text( + encoding="utf-8" + ) + svg_path = ROOT / "docs/assets/gdev-eval-lab-challenge-fail.svg" + svg = svg_path.read_text(encoding="utf-8") + + for path in ( + "docs/APPROVAL_FLOW.md", + "docs/evidence/GDEV_EVAL_FAILURE_PREVIEW_2026-07-13.md", + "docs/assets/gdev-eval-lab-challenge-fail.svg", + ): + assert path in readme + + assert "AgentService.needs_approval()" in approval + assert "RedisApprovalStore" in approval + assert "Atomic GETDEL" in approval + assert "tests/test_approval_flow.py" in approval + assert "implemented approval mechanics do not imply" in approval + + content_address = "sha256:656face21f27b496d4d3e8bb0b588824f5737d122c1275c710f3e5b15ff94b4b" + component_revision = "0e4c5f0fd50382bbf12ffd35cfca4632384fb0cc" + summary_hash = "d4bb0dec70d75de8d33d16f583a01ecbc17bf202ae04509d3dec63087a7b3a3b" + svg_hash = "e422da461ba3c133667d65dce3068cf65b689d9f98848c5c8184e9b89e051110" + + for claim in (content_address, component_revision, summary_hash): + assert claim in preview + assert claim in svg + assert content_address in evidence_index + assert "challenge gate is **FAIL**" in preview + assert "90 candidate HTTP calls" in preview + assert "10 labeled deterministic" in preview + assert "not a rerun" in preview + assert "--check" in preview + assert "Human review required" in svg + assert "Five cards show every failed threshold" in svg + assert hashlib.sha256(svg_path.read_bytes()).hexdigest() == svg_hash + + renderer = (ROOT / "scripts/render_eval_failure_preview.py").read_text(encoding="utf-8") + assert "manifest does not match the pinned canonical digest" in renderer + assert "manifest does not match the pinned content address" in renderer + assert "challenge-run.json does not match its manifest digest" in renderer + assert "this renderer only accepts a recorded failed gate" in renderer