From e09d2d1767ffcdecca531bef97fe0b5a78ce6de9 Mon Sep 17 00:00:00 2001 From: JumpMaster Date: Tue, 16 Jun 2026 02:02:37 -0600 Subject: [PATCH] docs(readme): architecture diagram, runnable demo, and the over-built note MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three presentation changes so the README lands as an architecture showcase: - Replace the ASCII architecture box-art with a GitHub-rendered mermaid flowchart of the webhook → outbox → Kafka → enrichment → diagnosis flow. - Add a "See it run" section: a copy-paste make compose-up → migrate → HMAC-signed webhook → poll path that produces a real diagnosis. The bare-stack output shows the evidence-citation gate doing its job — no verifiable evidence to cite, so it strips the invented refs, flags hallucinated_evidence, and caps confidence — with a pointer to the eval results for the full-context numbers. - Add a short, honest "isn't this over-built?" note backed by a new ADR 0015 (deliberate production patterns, and when to collapse Kafka into the outbox-as-queue). Demo commands verified live against the stack; the illustrative diagnosis JSON matches the real API response field-for-field. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 112 +++++++++++++----- .../0015-deliberate-production-patterns.md | 40 +++++++ 2 files changed, 121 insertions(+), 31 deletions(-) create mode 100644 docs/adr/0015-deliberate-production-patterns.md diff --git a/README.md b/README.md index 4a928e9..9a592cd 100644 --- a/README.md +++ b/README.md @@ -53,6 +53,62 @@ _From eval run `e7128643-f3f2-4101-9513-ad381fbd4b99` (auto-generated; do not ed > so it now measures consistency rather than a single lucky shot — two cases > where the model's category wobbles across shots are why it reads 80% here. +## See it run — webhook to diagnosis + +The numbers above come from the CI harness. To watch the live pipeline turn a +webhook into a diagnosis on your machine — first-time setup is `make bootstrap` +then `cp .env.example .env`, with `SENTINEL_ANTHROPIC_API_KEY` and +`SENTINEL_GENERIC_WEBHOOK_SECRET` filled in (the empty default disables the +generic source, so the webhook would 401): + +```bash +make compose-up # Postgres, Redis, Kafka, app — waits on healthchecks +make migrate # apply the schema (alembic upgrade head) + +# Fire a sample incident, HMAC-signed with your SENTINEL_GENERIC_WEBHOOK_SECRET: +SECRET=$(grep '^SENTINEL_GENERIC_WEBHOOK_SECRET=' .env | cut -d= -f2-) +BODY='{"id":"demo-1","service":"checkout-api","severity":"SEV1","title":"Checkout p99 latency spiked to 8s after deploy"}' +SIG=$(printf '%s' "$BODY" | openssl dgst -sha256 -hmac "$SECRET" | awk '{print $NF}') +curl -s -X POST localhost:8000/webhooks/generic \ + -H 'Content-Type: application/json' \ + -H "X-Sentinel-Signature: sha256=$SIG" \ + -d "$BODY" +# → {"status":"accepted","incident_id":""} + +# Diagnosis runs out-of-band (Kafka → enrichment → Anthropic). Poll the incident: +curl -s localhost:8000/incidents/ | jq '.diagnoses[0]' +``` + +The pipeline returns a schema-validated diagnosis — category, confidence, +hypothesis, suggested actions — with every evidence citation checked against the +assembled context first. On a bare stack with no deploy/log/alert integrations +wired, that check is exactly what you want to see fire: + +```jsonc +{ + "likely_category": "deploy", + "confidence": 0.35, // capped: no verifiable evidence to lean on + "hypothesis": "Checkout-api p99 latency spike to 8s is likely caused by a recent + deployment … though context status degradation prevents full verification.", + "hallucinated_evidence": true, // ← the evidence-citation gate caught invented refs + "evidence": [], // and stripped them before scoring + "suggested_actions": [ + { "description": "Roll back the most recent checkout-api deployment …", + "risk": "low", "requires_human_approval": true } + ] +} +``` + +That low, honest confidence is the system working as designed: with nothing real +to cite, the gate strips the invented references, flags `hallucinated_evidence`, +and caps confidence at 0.4 rather than letting the model sound sure. Point the +fetchers at real tooling — or see the eval results above, scored against real +postmortems with full context — and the same gate rewards well-grounded +diagnoses with high confidence instead. + +> The evals replay recorded LLM responses and cost nothing; this live path makes +> a real Anthropic call (a few cents per diagnosis). + ## Why it's built this way A few load-bearing choices that distinguish Sentinel from a typical "LLM wrapper": @@ -66,44 +122,37 @@ A few load-bearing choices that distinguish Sentinel from a typical "LLM wrapper ADRs for the non-obvious calls live in [`docs/adr/`](docs/adr/). +> **Isn't this over-built for a single-tenant copilot?** For the workload, yes — +> and deliberately so. The point is to demonstrate distributed-systems patterns +> built correctly (transactional outbox, idempotent Kafka producer, per-dependency +> circuit breakers, pgvector memory), not to ship the smallest thing that works. +> A real single-tenant deployment would collapse Kafka into the outbox-as-queue +> and drop the broker, keeping everything else — see +> [ADR 0015](docs/adr/0015-deliberate-production-patterns.md). Naming the +> trade-off is part of the engineering. + ## Stack Python 3.12 · FastAPI · asyncio · Pydantic v2 · Postgres 16 + `pgvector` · Redis 7 · Kafka (KRaft) · Anthropic (`claude-sonnet-4-5` default) · OpenTelemetry + Prometheus + structlog · Docker Compose. Next.js 14 + Tailwind for the demo UI (later phase). ## Architecture +```mermaid +flowchart TD + A["POST /webhooks/:source
HMAC verified · payload-hash idempotent
normalized to NormalizedAlert"] + A -->|"202 Accepted, returns fast"| B[("Postgres
incidents + outbox_events
written in one transaction")] + B -->|"outbox drainer"| K{{"Kafka · sentinel.incidents"}} + K -->|"incident.opened"| E["Enrichment
parallel fetchers · per-integration circuit breakers
deploys · related/active alerts · recent logs · runbooks"] + E <-->|"similar incidents"| MEM[("pgvector
incident memory")] + E -->|"incident.enriched"| K + K -->|"incident.enriched"| D["Diagnosis
versioned prompt to Anthropic at temperature 0
schema-validated · evidence-citation gate
idempotent per incident + prompt_version"] + D --> R["Structured diagnosis
confidence · evidence refs · suggested actions"] + RES["POST /incidents/:id/resolve"] -.->|"embed root cause on resolve"| MEM ``` - ┌──────────────────────────────────────────────────────────┐ - │ POST /webhooks/{source} (HMAC verified, payload-hash │ - │ idempotent, normalized → NormalizedAlert) │ - └───────────────┬──────────────────────────────────────────┘ - │ 202 Accepted (returns fast) - ▼ - ┌──────────────────────────────────────────────────────────┐ - │ Postgres ←→ outbox_events │ - │ incidents (fingerprint dedup, event_log appended) │ - └───────────────┬──────────────────────────────────────────┘ - │ outbox drainer - ▼ - ┌──────────────────────────────────────────────────────────┐ - │ Kafka topic: sentinel.incidents │ - │ incident.opened / incident.recurred / incident.enriched│ - └───────────────┬──────────────────────────────────────────┘ - ▼ - ┌──────────────────────────────────────────────────────────┐ - │ Enrichment worker — parallel fetchers behind │ - │ per-integration circuit breakers │ - │ (deploys · related alerts · active alerts · recent logs │ - │ · runbooks · similar incidents from pgvector) │ - │ → IncidentContext written back, incident.enriched event │ - └───────────────┬──────────────────────────────────────────┘ - ▼ - ┌──────────────────────────────────────────────────────────┐ - │ Diagnosis worker — versioned prompt + Anthropic call, │ - │ schema-validated response, evidence-citation gate, │ - │ idempotent persistence keyed on (incident, prompt_ver) │ - └──────────────────────────────────────────────────────────┘ -``` + +Every external call in that flow has a timeout and a circuit breaker; every LLM +response is schema-validated; every evidence citation is verified against the +assembled context before the diagnosis is trusted. ## HTTP API @@ -181,6 +230,7 @@ If a code path can't articulate its failure mode, it isn't done. - Python 3.12 - Docker + Docker Compose v2 (Postgres 16 + pgvector, Redis 7, Kafka 3.7) - GNU Make +- `jq` and `openssl` — only for the "See it run" demo above (both ship with most systems) ### Bootstrap diff --git a/docs/adr/0015-deliberate-production-patterns.md b/docs/adr/0015-deliberate-production-patterns.md new file mode 100644 index 0000000..7026717 --- /dev/null +++ b/docs/adr/0015-deliberate-production-patterns.md @@ -0,0 +1,40 @@ +# ADR 0015 — Deliberately built to full production patterns (and when to collapse them) + +## Status + +Accepted, 2026-06-16. Revisit if Sentinel ever targets a real single-tenant deployment rather than serving as a portfolio demonstration of distributed-systems patterns. + +## Context + +Sentinel is a public portfolio repository held to a production engineering bar. A fair critique of the architecture — for the *workload it actually serves* — is that it is over-built: a single-tenant, single-shot diagnosis copilot at low incident volume does not strictly need a Kafka broker, a transactional outbox, and a separate consumer group. The same correctness could be had with a simpler topology. + +That critique is correct on the merits and worth stating plainly rather than hiding. The reason the apparatus exists anyway is that the repository's purpose is to **demonstrate that these patterns can be designed and built correctly**, not to ship the minimal system that solves the immediate problem. Breadth-of-production-patterns-executed-correctly is the deliverable here, so the apparatus is a feature, not an accident. + +This ADR records that trade-off explicitly so the choice reads as judgment rather than gold-plating, and so a future reader knows exactly what to remove if they ever repurpose this for a real low-volume deployment. + +## Decision + +V1 deliberately implements full production patterns even where a smaller system would suffice: + +- **Transactional outbox + Kafka** for webhook → out-of-band diagnosis (ADR 0001, ADR 0002), rather than running diagnosis inline in the request path or polling a table. +- **Per-dependency circuit breakers + timeouts** on every fetcher (invariant 1), rather than bare `try/except`. +- **pgvector incident memory** with embed-on-resolve (ADR 0005), rather than keyword search over past incidents. +- **A statistically-rigorous eval harness** with paired-bootstrap regression gating, rather than spot-checking a few prompts. + +These are kept because demonstrating them — correctly, with ADRs and tests — is the point. + +## Consequences + +**The honest simplification.** A real single-tenant deployment at low incident volume would collapse this topology: + +- **Drop the Kafka broker; use the outbox table as the queue.** The `outbox_events` table already exists and is written in the same transaction as the incident (ADR 0001). A `SELECT ... FOR UPDATE SKIP LOCKED` poller over that table gives the same at-least-once, ordered-per-incident delivery without a broker to operate. The consumer classes (`EnrichmentConsumer`, `DiagnosisConsumer`) are already self-contained (ADR 0004), so the change is localized to the transport, not the processing logic. +- **Keep everything else.** Circuit breakers, the evidence-citation gate, idempotency, embed-on-resolve, and the eval harness are not workload-dependent — they are correctness and quality machinery that earns its keep at any scale. + +The result would be a Postgres + Redis + FastAPI deployment with no broker: materially simpler to operate, with the same correctness guarantees. This is the deployment a real single-tenant user should run. + +**What this ADR is not.** It is not a deprecation. V1 keeps the full topology on purpose. It is a written acknowledgement of the seam so that the "isn't this over-built?" question has a documented, considered answer instead of a defensive one. + +## Triggers to revisit + +- The repository is forked for an actual single-tenant deployment (collapse Kafka into the outbox-as-queue per above). +- Multi-tenancy or cross-service fan-out is introduced (the broker earns its place; this ADR no longer applies).