AI Inference Reliability Platform

A Datadog-lite observability and incident detection system for distributed inference workloads. Scrapes metrics, detects anomalies, correlates cascading failures across services using a dependency graph, generates LLM-powered root-cause summaries, and autoscales under load.

Architecture

gateway (TypeScript/Express)     inference (Java/Spring Boot)
       │         │                        │
       │    /api/predict ──────────► /predict
       │                                  │
       └──── /metrics ────┐  ┌── /metrics ┘
                          ▼  ▼
                  collector (Python/FastAPI)
                     │    │    │       │
                     ▼    ▼    ▼       ▼
                  Kafka  Redis Detector Dependency Graph
                    │      │       │       │
                    ▼      │       ▼       ▼
               PostgreSQL  │   Cascading Failure Detection
                    │      │       │
                    ▼      ▼       ├──► Gemini API (LLM root-cause summary)
              Gateway Dashboard    ▼
              API ◄── PostgreSQL + Kafka (incidents)

Services

Service	Language	Port	Role
gateway	TypeScript / Express	3000	Request intake, predict proxy to inference, Prometheus metrics, dashboard API
inference	Java 21 / Spring Boot	8080	Model serving simulator, chaos injection endpoints
collector	Python / FastAPI	8000	Metrics scraping, Kafka streaming, incident detection, cascading failure correlation, LLM root-cause summaries

Infrastructure

Component	Image	Purpose
Kafka	apache/kafka:3.9.0	Event streaming (KRaft mode, no ZooKeeper)
PostgreSQL	bitnami/postgresql	Historical metrics and incident storage
Redis	redis:7-alpine	Live service state cache

Quick Start — Local (Docker Compose)

docker compose up --build

Services available at localhost:3000, localhost:8080, localhost:8000. No Kafka/Redis/PostgreSQL in compose — local mode runs health endpoints only.

Deploy to GKE

Prerequisites

• gcloud CLI authenticated
• GKE Autopilot cluster created
• Artifact Registry repo created
• kubectl connected to cluster
• Helm installed

1. Build and push images

# Set your project/region
PROJECT=ai-inference-497423
REGION=us-south1
REPO=inference-repo
REGISTRY=$REGION-docker.pkg.dev/$PROJECT/$REPO

# Configure Docker auth
gcloud auth configure-docker $REGION-docker.pkg.dev

# Build and push all three
for svc in gateway inference collector; do
  docker build -t $REGISTRY/$svc:latest services/$svc
  docker push $REGISTRY/$svc:latest
done

2. Deploy infrastructure

# PostgreSQL
helm install postgresql bitnami/postgresql -f k8s/postgresql/values.yaml

# Apply schema
kubectl exec -i postgresql-0 -- env PGPASSWORD=postgres \
  psql -U postgres -d inference_platform < k8s/postgresql/schema.sql

# Kafka + Redis
kubectl apply -f k8s/kafka/kafka.yaml
kubectl apply -f k8s/redis/redis.yaml

# Wait for infrastructure
kubectl get pods -w  # wait until all Running

3. Deploy application

kubectl apply -f k8s/base/

# Optionally, create the LLM API key secret for root-cause summaries
kubectl create secret generic llm-api-key --from-literal=api-key=YOUR_GEMINI_API_KEY

4. Verify

# All pods running
kubectl get pods

# Port-forward to test
kubectl port-forward svc/gateway 3000:3000

# Live service status (from Redis)
curl http://localhost:3000/api/services/status

# Historical data (from PostgreSQL)
curl "http://localhost:3000/api/services/gateway/history?range=1h"

# Active incidents
curl http://localhost:3000/api/incidents/active

Dashboard API

Endpoint	Source	Description
GET /api/services/status	Redis	Live status of all services
GET /api/services/:name/history?range=1h	PostgreSQL	Historical health checks (1h, 6h, 24h, 7d)
GET /api/incidents	PostgreSQL	All incidents, newest first (limit 50)
GET /api/incidents/active	PostgreSQL	Unresolved incidents
POST /api/predict	Proxy → inference	Forwards to inference /predict, returns 502 when inference is down

All endpoints return { data, error, timestamp }.

Chaos Testing

The inference service supports fault injection:

# Enable chaos (50% errors on /predict, 503 on /ready)
kubectl exec <inference-pod> -- wget -qO- --post-data="" http://localhost:8080/chaos/enable

# Add artificial latency
kubectl exec <inference-pod> -- wget -qO- http://localhost:8080/chaos/latency?ms=800

# Disable
kubectl exec <inference-pod> -- wget -qO- --post-data="" http://localhost:8080/chaos/disable

Or run the full demo:

bash scripts/demo-chaos.sh

The collector detects sustained failures within ~20 seconds and auto-creates incidents. When multiple services degrade within 30 seconds, cascading failure detection identifies the root cause via the dependency graph. If a Gemini API key is configured, an LLM-generated root-cause summary is attached to the incident. When services recover, incidents are auto-resolved.

Incident Detection Rules

Rule	Threshold	Sustained	Incident Type
High error rate	> 5%	2+ checks (20s)	high_error_rate
High latency	p95 > 500ms	2+ checks (20s)	high_latency
Service down	Unreachable	2+ checks (20s)	high_error_rate
Cascading failure	2+ services degraded within 30s	Dependency graph validated	cascading_failure

Autoscaling (HPA)

Service	Min Replicas	Max Replicas	Scale Trigger
gateway	2	8	CPU > 60%
inference	3	10	CPU > 50%

Project Structure

services/
  gateway/          TypeScript/Express — predict proxy, dashboard API, Prometheus metrics
  inference/        Java 21/Spring Boot — model simulator, chaos endpoints
  collector/        Python/FastAPI — scraper, Kafka producer/consumer, detector,
                      cascading failure correlation, LLM root-cause summaries
config/
  dependencies.yml  Service dependency graph for cascading failure detection
k8s/
  base/             Service deployments, ConfigMaps, Ingress, HPAs
  kafka/            Kafka StatefulSet (apache/kafka, KRaft)
  redis/            Redis StatefulSet
  postgresql/       Helm values + SQL schema
scripts/
  demo-chaos.sh     End-to-end cascading failure demo
docs/
  progress/         Layer-by-layer build progress files
  interview-prep.md Detailed interview preparation guide

Build Layers

Layer	What
0	Dockerized microservices with health endpoints
1	Kubernetes deployment with probes and Ingress
2	Kafka + PostgreSQL metrics streaming pipeline
3	Redis live state + dashboard API
4	Incident detection + chaos testing
5	Cascading failure detection + LLM root-cause + HPA

Cleanup

# Delete application
kubectl delete -f k8s/base/

# Delete infrastructure
kubectl delete -f k8s/kafka/kafka.yaml
kubectl delete -f k8s/redis/redis.yaml
helm uninstall postgresql

# Delete PVCs
kubectl delete pvc --all

# Delete GKE cluster (stops billing)
gcloud container clusters delete inference-cluster --region us-south1