AI on-call copilot. Sentinel ingests production alerts (Sentry, PagerDuty, Datadog, generic webhooks), assembles incident context in parallel from your existing tooling, and produces a structured diagnosis — with confidence scoring, evidence citations against the supplied context, and suggested remediations. Single-shot, schema-validated, and run at temperature 0. Learns from resolutions via a pgvector incident memory.
Public engineering portfolio. Held to a production bar — every external call has a timeout and a circuit breaker, every LLM response is schema-validated, every evidence citation is verified, and the eval harness blocks regressions in CI.
From eval run e7128643-f3f2-4101-9513-ad381fbd4b99 (auto-generated; do not edit between markers).
| Metric | Value |
|---|---|
| category_match | 0.90 |
| hypothesis_cosine | 0.82 |
| action_coverage | 0.75 |
| evidence_quality | 1.00 |
| Case | category | hypothesis | actions | evidence | stability |
|---|---|---|---|---|---|
| atlassian-2022-04-04-deletion | 1.00 | 0.86 | 0.72 | 1.00 | 0.02 |
| aws-2021-12-07-useast1 | 1.00 | 0.86 | 0.77 | 1.00 | 0.00 |
| cloudflare-2019-07-02-regex | 1.00 | 0.81 | 0.78 | 1.00 | 0.02 |
| cloudflare-2022-06-21-bgp | 1.00 | 0.87 | 0.76 | 1.00 | 0.01 |
| fastly-2021-06-08-config | 1.00 | 0.76 | 0.82 | 1.00 | 0.01 |
| github-2018-10-21-network | 0.33 | 0.77 | 0.74 | 1.00 | 0.15 |
| gitlab-2017-01-31-db-deletion | 1.00 | 0.79 | 0.72 | 1.00 | 0.00 |
| roblox-2021-10-28-consul | 0.67 | 0.80 | 0.75 | 1.00 | 0.16 |
| slack-2021-01-04-dns | 1.00 | 0.80 | 0.70 | 1.00 | 0.01 |
| stripe-2019-07-10-db-failover | 1.00 | 0.84 | 0.77 | 1.00 | 0.00 |
- pass_rate_strict: 80.0%
- mean_stability: 0.039
Three shots per case. Shot 0 runs the full webhook → diagnose pipeline and is the persisted, scored shot; shots 1+ re-run diagnosis in-memory against the same context to measure run-to-run LLM stability (the
stabilitycolumn is the mean per-case stddev across shots — lower is steadier). See ADR 0009 for why stability is measured at the LLM-call level, decoupled from the production idempotency contract, and ADR 0007 for the (kind, id) evidence-match contract these numbers rely on.
pass_rate_strictcounts a case only when every shot clears all four thresholds (category exact, hypothesis ≥ 0.7, actions ≥ 0.6, evidence ≥ 0.8), so it now measures consistency rather than a single lucky shot — two cases where the model's category wobbles across shots are why it reads 80% here.
The numbers above come from the CI harness. To watch the live pipeline turn a
webhook into a diagnosis on your machine — first-time setup is make bootstrap
then cp .env.example .env, with SENTINEL_ANTHROPIC_API_KEY and
SENTINEL_GENERIC_WEBHOOK_SECRET filled in (the empty default disables the
generic source, so the webhook would 401):
make compose-up # Postgres, Redis, Kafka, app — waits on healthchecks
make migrate # apply the schema (alembic upgrade head)
# Fire a sample incident, HMAC-signed with your SENTINEL_GENERIC_WEBHOOK_SECRET:
SECRET=$(grep '^SENTINEL_GENERIC_WEBHOOK_SECRET=' .env | cut -d= -f2-)
BODY='{"id":"demo-1","service":"checkout-api","severity":"SEV1","title":"Checkout p99 latency spiked to 8s after deploy"}'
SIG=$(printf '%s' "$BODY" | openssl dgst -sha256 -hmac "$SECRET" | awk '{print $NF}')
curl -s -X POST localhost:8000/webhooks/generic \
-H 'Content-Type: application/json' \
-H "X-Sentinel-Signature: sha256=$SIG" \
-d "$BODY"
# → {"status":"accepted","incident_id":"<uuid>"}
# Diagnosis runs out-of-band (Kafka → enrichment → Anthropic). Poll the incident:
curl -s localhost:8000/incidents/<uuid> | jq '.diagnoses[0]'The pipeline returns a schema-validated diagnosis — category, confidence, hypothesis, suggested actions — with every evidence citation checked against the assembled context first. On a bare stack with no deploy/log/alert integrations wired, that check is exactly what you want to see fire:
That low, honest confidence is the system working as designed: with nothing real
to cite, the gate strips the invented references, flags hallucinated_evidence,
and caps confidence at 0.4 rather than letting the model sound sure. Point the
fetchers at real tooling — or see the eval results above, scored against real
postmortems with full context — and the same gate rewards well-grounded
diagnoses with high confidence instead.
The evals replay recorded LLM responses and cost nothing; this live path makes a real Anthropic call (a few cents per diagnosis).
A few load-bearing choices that distinguish Sentinel from a typical "LLM wrapper":
- Single-shot diagnosis, not agentic. Enrichment pre-assembles all context in parallel; the LLM only reasons over what it was given. Predictable latency, deterministic evals, bounded cost. No tool-calling loops in V1.
- Webhook → Kafka → out-of-band diagnosis.
POST /webhooks/{source}returns202immediately after persisting the incident and emittingincident.openedvia a transactional outbox. Diagnosis runs off the event, never in the request path. - Fingerprint dedup, not external-ID dedup.
sha256(service ‖ normalized_title ‖ severity). Same fingerprint within 1h updates the existing incident and appends to the event log; it does not create a new row. Raw-payload idempotency (SETNX webhook:{source}:{sha256(body)}, 24h TTL) defends against retried-with-changes. - Evidence-citation gate. After the LLM returns, every
EvidenceRef.idmust resolve to an item that was actually in the assembled context. Hallucinated citation →hallucinated_evidence: true, confidence capped at 0.4, metric incremented. This is the project's headline quality signal. - Embedding on resolve, not on open. Similar-incident retrieval improves once embeddings reflect actual root cause, not surface symptom. The "gets smarter from use" loop.
- Monolith with strict module boundaries. Inter-module calls go through repository or service interfaces; non-
persistence/modules never touch the DB directly. The API process owns both the enrichment and diagnosis Kafka consumers — horizontal scale is N API replicas sharing a consumer group, partition count caps concurrency. Splittable into services later without rewriting.
ADRs for the non-obvious calls live in docs/adr/.
Isn't this over-built for a single-tenant copilot? For the workload, yes — and deliberately so. The point is to demonstrate distributed-systems patterns built correctly (transactional outbox, idempotent Kafka producer, per-dependency circuit breakers, pgvector memory), not to ship the smallest thing that works. A real single-tenant deployment would collapse Kafka into the outbox-as-queue and drop the broker, keeping everything else — see ADR 0015. Naming the trade-off is part of the engineering.
Python 3.12 · FastAPI · asyncio · Pydantic v2 · Postgres 16 + pgvector · Redis 7 · Kafka (KRaft) · Anthropic (claude-sonnet-4-5 default) · OpenTelemetry + Prometheus + structlog · Docker Compose. Next.js 14 + Tailwind for the demo UI (later phase).
flowchart TD
A["POST /webhooks/:source<br/>HMAC verified · payload-hash idempotent<br/>normalized to NormalizedAlert"]
A -->|"202 Accepted, returns fast"| B[("Postgres<br/>incidents + outbox_events<br/>written in one transaction")]
B -->|"outbox drainer"| K{{"Kafka · sentinel.incidents"}}
K -->|"incident.opened"| E["Enrichment<br/>parallel fetchers · per-integration circuit breakers<br/>deploys · related/active alerts · recent logs · runbooks"]
E <-->|"similar incidents"| MEM[("pgvector<br/>incident memory")]
E -->|"incident.enriched"| K
K -->|"incident.enriched"| D["Diagnosis<br/>versioned prompt to Anthropic at temperature 0<br/>schema-validated · evidence-citation gate<br/>idempotent per incident + prompt_version"]
D --> R["Structured diagnosis<br/>confidence · evidence refs · suggested actions"]
RES["POST /incidents/:id/resolve"] -.->|"embed root cause on resolve"| MEM
Every external call in that flow has a timeout and a circuit breaker; every LLM response is schema-validated; every evidence citation is verified against the assembled context before the diagnosis is trusted.
Pydantic-validated request and response on every boundary; the OpenAPI schema is publishable (make openapi). Shipped (PRs #58 / #59 / #60):
| Method | Path | |
|---|---|---|
POST |
/webhooks/{source} |
ingest — HMAC + payload-hash idempotent; returns 202 |
POST |
/incidents |
manual create through the fingerprint + ingest path |
GET |
/incidents |
list with status / service / severity filters + pagination |
GET |
/incidents/{id} |
detail incl. assembled context + latest diagnosis |
POST |
/incidents/{id}/resolve |
submit resolution (triggers embedding refresh on resolve) |
POST |
/incidents/{id}/diagnose |
re-run diagnosis — replay-backed, suggest-only |
GET |
/evals/runs · /evals/runs/{id} · /evals/baseline |
eval-run history + current baseline |
GET |
/metrics |
Prometheus exposition |
GET |
/healthz · /readyz |
liveness; readiness (Postgres + Redis probes + consumer aliveness) |
Realtime SSE (/incidents/{id}/stream) and WebSocket (/ws/incidents) are the remaining slice (Work Area I, PR 3).
sentinel/
├── ingestion/ webhook receivers, HMAC verify, fingerprinting, idempotency, outbox drainer
├── integrations/ sentry · pagerduty · datadog · generic — adapters normalize to NormalizedAlert
├── enrichment/ parallel fetchers + circuit breaker, orchestrator, Kafka consumer
├── diagnosis/ LLM client, versioned prompt bundle, validation gate, persistence, consumer
├── memory/ pgvector incident store + embedding pipeline, embed-on-resolve feedback
├── persistence/ SQLAlchemy models, Alembic migrations, repositories
├── observability/ Prometheus metrics, OTel tracing, structlog, LLM audit log + cost meter
├── schemas/ Pydantic v2 contracts shared across modules (NormalizedAlert, Diagnosis, …)
├── api/ FastAPI app — routes: health, webhooks, incidents, resolve, diagnose, evals, metrics
├── evals/ eval harness: postmortem corpus, scoring, multi-shot stability, CI gate
└── config/ pydantic-settings, per-env defaults, secret loading
Built work area by work area, each behind a green CI gate. The eval harness (K) was prioritized ahead of the API surface (I); the REST surface has since landed (PRs #58 / #59 / #60), leaving realtime (SSE/WS) and the UI (L).
| Phase | Work area | Status |
|---|---|---|
| 1 | Foundation & tooling (A) — Dockerfile, compose, Makefile, ruff/mypy/pytest, pre-commit, CI | ✅ Landed |
| 2 | Persistence + core schemas + observability skeleton (B, C, J) — Alembic, models, metrics, OTel, structlog | ✅ Landed |
| 3 | Webhook ingestion + integration adapters (D, E) — /webhooks/{source}, HMAC, fingerprinting, dedup, transactional outbox |
✅ Landed |
| 4 | Enrichment pipeline (F) — parallel fetchers, circuit breakers, incident.enriched event |
✅ Landed |
| 5 | Diagnosis agent (G) — versioned prompt, Anthropic call, schema + evidence gate, idempotent persistence | ✅ Landed |
| 6 | Memory / feedback loop (H) — embedding on resolve, similar-incident retrieval feedback | ✅ Landed |
| 7 | API surface (I) — REST (incidents CRUD/list/detail, re-diagnose, /metrics, /readyz PG+Redis probes, evals reads), publishable OpenAPI ✅ landed; realtime SSE/WS ⏳ |
🔄 REST landed |
| 8 | Eval harness (K) — postmortem corpus, scoring, multi-shot stability, CI gate + nightly full | ✅ Landed |
| 9 | UI (L) — Next.js incident view | ⏳ Planned |
| 10 | Load + chaos (M) — Locust ingestion load, Toxiproxy Redis-outage + in-process breaker fault injection | ✅ Landed |
Designs and per-phase plans live in plans/.
The non-negotiable invariants. Any change that weakens one needs an ADR.
- Every external call has a timeout and a circuit breaker (
5 failures / 60s → open, 30s → half-open, 1 trial → closed). Failures are logged and counted, never raised — fetchers returnFetcherResult{status: ok | degraded | failed}. - Every LLM response is Pydantic-validated. Invalid → retry once → fail loud.
- Every evidence citation is verified against the supplied context. Hallucinated →
hallucinated_evidence: true, confidence capped at 0.4,sentinel_hallucinated_evidence_rateincremented. - Context items use stable, checkable IDs in the prompt (
[deploy:abc123],[similar:incident-uuid]); the validation gate depends on this format. - Confidence rubric is fixed (0.0–0.3 speculation · 0.4–0.6 plausible · 0.7–0.85 strong · 0.86–1.0 direct causal). Change → bump
prompt_versionand re-baseline evals. - Pydantic on every API boundary. No
dict[str, Any]in request/response models. OpenAPI must be publishable. - Suggest only, never execute.
SuggestedAction.requires_human_approvaldefaults to true. No code path executes remediations. - Every webhook is idempotent on payload hash. Every migration is reversible (
upgradeanddowngrade). - Every PR includes tests. CI green or it doesn't merge.
If a code path can't articulate its failure mode, it isn't done.
- Python 3.12
- Docker + Docker Compose v2 (Postgres 16 + pgvector, Redis 7, Kafka 3.7)
- GNU Make
jqandopenssl— only for the "See it run" demo above (both ship with most systems)
make bootstrap # create .venv, install runtime + dev deps, install pre-commit
cp .env.example .env # fill in SENTINEL_ANTHROPIC_API_KEY and per-source webhook secrets
make compose-up # boot Postgres, Redis, Kafka, app (waits on healthchecks)
curl localhost:8000/healthz # → {"status":"ok"}
make compose-down # tear down (-v removes volumes)make fmt # ruff format + autofix
make lint # ruff format --check + ruff check (CI parity)
make typecheck # mypy --strict
make test # unit tests
make test-integration # integration tests (needs `make compose-up`)
make migrate # alembic upgrade head
make migrate-down # alembic downgrade -1
make evals-smoke # 5-case smoke set (cassette replay)
make evals # full corpus (cassette replay)
make readme-numbers # patch README from the latest eval run
make load-smoke # concurrent-ingestion invariants (zero-drop + p95); needs docker
make load # locust 100 req/s x 5min, consumers-off stack (creditless)
make chaos # resilience: breaker fault-injection + toxiproxy Redis outage
make openapi # export the publishable OpenAPI schema to openapi.jsonmake load and make chaos never call the Anthropic API: load runs the stack
with the diagnosis + memory consumers disabled, and chaos does the same for its
infra-outage tests (B1) or injects faults entirely in-process (B2). See
ADR 0010.
Single test:
pytest tests/unit/diagnosis/test_validation.py::test_hallucinated_evidence_caps_confidenceEvery PR runs: lint → typecheck → unit → integration → evals-gate (full corpus, cassette replay + regression gate). A nightly workflow runs the full corpus against the live Anthropic API and opens a drift issue on regression. README numbers come from evals/results/<latest>.md — never hand-edited.
All settings are loaded by sentinel.config.settings.Settings via pydantic-settings. Source is environment variables prefixed SENTINEL_. See .env.example for the full list — runtime mode, Postgres DSN, Redis URL, Kafka brokers + topic, Anthropic key + model, per-source webhook HMAC secrets, diagnosis token caps and timeout, OTel endpoint, LLM audit log path.
0001-transactional-outbox.md— Atomic Postgres + Kafka via anoutbox_eventstable and a drainer.0002-enrichment-same-topic-roundtrip.md—incident.enrichedreuses the samesentinel.incidentstopic.0003-single-llm-provider-no-breaker.md— V1 is Anthropic-only; timeout + retry-once-then-fail substitutes for a per-LLM breaker.
Multi-tenancy. Hosted auth (local / reverse-proxy only). Automatic remediation execution. Real-time log search at scale. Multiple LLM providers (interface stays single-impl).
If you find this project useful, you can support its development:
{ "likely_category": "deploy", "confidence": 0.35, // capped: no verifiable evidence to lean on "hypothesis": "Checkout-api p99 latency spike to 8s is likely caused by a recent deployment … though context status degradation prevents full verification.", "hallucinated_evidence": true, // ← the evidence-citation gate caught invented refs "evidence": [], // and stripped them before scoring "suggested_actions": [ { "description": "Roll back the most recent checkout-api deployment …", "risk": "low", "requires_human_approval": true } ] }