Autonomous Data Platform Runtime

Why I Built This

I built this to model the operating layer above observability: triage, blast radius, root cause prediction, governed remediation, and executive/operator briefings.

The key challenge I wanted to capture was the part that usually gets hidden in simple demos: how data, signals, decisions, constraints, evidence, and operating risk move through a system that someone else could inspect and run locally.

I intentionally kept this version local and synthetic because the goal is to make the architecture and tradeoffs reviewable without external services, private data, paid APIs, or cloud setup.

Real Business Problem

Large data and AI platforms emit fragmented reliability, quality, governance, model, semantic, and RAG signals; operators need decisions, not more dashboards.

This matters because production teams do not only need outputs. They need evidence, ownership, repeatable validation, failure modes, and a path from local prototype to governed production system.

What This Project Proves

• platform reliability
• incident intelligence
• root-cause reasoning
• governance-aware remediation
• operator simulation
• scorecard reporting
• production-style data pipeline design
• synthetic but realistic data modeling
• scorecard generation
• API/dashboard serving
• testable architecture
• honest limitation framing

Architecture In Plain English

Synthetic platform signals are correlated into incidents, scored for impact, matched to root-cause patterns, routed through remediation policy, and summarized in briefings and scorecards.

The important pattern is that inputs are not just transformed into outputs. They are turned into scored, documented artifacts that can be reviewed by operators, analysts, engineers, and business stakeholders.

Key Design Decisions

• Synthetic data keeps the repo safe to run and share publicly.
• Deterministic local logic makes validation repeatable without paid APIs.
• DuckDB or local artifacts provide warehouse-style inspection without cloud setup.
• FastAPI shows how the system could be served as a service layer.
• Streamlit gives reviewers a fast way to inspect the outputs visually.
• Scorecards make quality, risk, reliability, or readiness measurable.
• Tests and Ruff keep the repo from being only documentation.
• Docker/CI files show the intended deployment shape without claiming production readiness.

See docs/design-decisions.md for the detailed tradeoff record.

Validation Evidence

Latest validation run: 2026-06-02.

• Pipeline: passed
• Pytest: passed (65 tests)
• Ruff: passed
• Repository quality docs check: passed
• Detailed command output is recorded in docs/validation-log.md.

Generated Artifacts To Inspect

• platform signals
• incident records
• blast-radius analysis
• root-cause reports
• remediation plans
• operator actions
• executive briefings

How To Review This Repo

Recruiter / hiring manager:

• Read this README first.
• Review docs/recruiter-summary.md if present.
• Check docs/validation-log.md.
• Use docs/repo-review-guide.md for the quickest path.

Senior engineer:

• Review the architecture docs.
• Inspect the src/ modules.
• Inspect tests and generated scorecards.
• Read docs/design-decisions.md and docs/tradeoffs-and-simplifications.md.

Interview path:

• Run the pipeline command from the validation log.
• Launch the dashboard or API if this repo includes them.
• Explain one design decision and one simplification honestly.

Known Limitations

• Synthetic data only.
• Local prototype rather than deployed production system.
• Deterministic rules or simulations where a production system may use live models, streaming data, or enterprise integrations.
• No real sensitive data is used.
• No authentication, RBAC, secrets management, or production security boundary unless explicitly stated elsewhere in the repo.
• External systems are simulated instead of connected live.

Production Roadmap

• ingest OpenLineage/Datadog/PagerDuty signals
• add live workflow orchestration
• integrate approval systems
• connect warehouse/lakehouse
• add observability and RBAC

See docs/production-roadmap.md for the staged roadmap.

Executive Summary

This project simulates the next layer of enterprise data and AI platforms: autonomous operations.

A traditional observability platform asks: "What is broken?"

This project asks: "What is broken, why did it happen, what is the blast radius, what should we do next, and how confident are we that the recovery action is safe?"

This project demonstrates autonomous data platform operations: turning fragmented signals into root-cause analysis, governed remediation decisions, and executive-ready incident intelligence.

Business Problem

Enterprise data platforms are becoming too complex for manual operations alone. Teams face thousands of pipelines, many data products, AI systems, semantic metrics, governance policies, SLA misses, model drift, schema drift, policy violations, hallucination alerts, and downstream business incidents.

The issue is no longer just monitoring. The challenge is decision-making.

Project Goal

Build a production-style local autonomous data platform runtime that ingests synthetic platform signals, detects and correlates incidents, estimates blast radius, predicts probable root causes, recommends remediation actions, evaluates governance constraints, scores recovery confidence, simulates autonomous operators, uses historical incident memory, and generates executive/operator briefings.

Architecture

flowchart LR
    A["Synthetic Platform Signals"] --> B["Incident Triage"]
    C["Synthetic Incidents"] --> B
    D["Historical Incident Memory"] --> E["Root-Cause Engine"]
    B --> F["Blast Radius Analysis"]
    F --> E
    E --> G["Remediation Recommender"]
    H["Governance Action Policies"] --> G
    G --> I["Autonomous Operators"]
    I --> J["Recovery Confidence + Action History"]
    J --> K["Executive / Operator Briefings"]
    K --> L["DuckDB Runtime Warehouse"]
    L --> M["FastAPI + Streamlit"]

Loading

Runtime Flow

flowchart TD
    A["Generate Signals"] --> B["Generate Incidents"]
    B --> C["Generate Historical Memory"]
    C --> D["Normalize Signals"]
    D --> E["Triage Incidents"]
    E --> F["Calculate Blast Radius"]
    F --> G["Predict Root Cause"]
    G --> H["Recommend Remediation"]
    H --> I["Enforce Action Policy"]
    I --> J["Simulate Operators"]
    J --> K["Forecast Stability"]
    K --> L["Briefings + Scorecards"]

Loading

Evidence Generated

• blast_radius_analysis.json/csv
• root_cause_prediction_report.json/csv
• remediation_recommendations.csv
• autonomous_operator_actions.csv
• operator_decision_history.json
• platform_stability_forecast.json/csv
• autonomous_runtime_scorecard.json/csv
• platform_recovery_scorecard.json/csv
• executive_incident_briefings.md
• operator_incident_briefings.md

How To Run

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

python -m src.data_generation.generate_platform_signals
python -m src.data_generation.generate_incidents
python -m src.data_generation.generate_incident_memory
python -m src.pipeline.run_all
python -m pytest
python -m ruff check .

streamlit run src/dashboard/app.py
uvicorn src.api.main:app --reload

API

Endpoints include /health, /runtime-summary, /incidents, /blast-radius/{incident_id}, /root-cause/{incident_id}, /remediation/{incident_id}, /operator-actions, /platform-stability, /executive-briefings, /scorecards, /simulate-incident, /recommend-remediation, and /simulate-operator-action.

Known Limitations

• Synthetic signals only
• Deterministic rules instead of live LLM agents
• Local DuckDB instead of enterprise warehouse
• Simulated integrations instead of real platform APIs
• No cloud deployment
• No authentication
• No live pager/alerting integration
• No OpenLineage, MLflow, Datadog, or PagerDuty integration yet

Future Enhancements

• LLM-assisted operator reasoning
• LangGraph/AutoGen/CrewAI operator workflow
• OpenLineage/Marquez integration
• MLflow model registry integration
• Datadog/Prometheus/Grafana ingestion
• PagerDuty/Slack alert routing
• Kafka streaming signal ingestion
• Airflow DAG remediation hooks
• Snowflake/Databricks deployment
• OpenPolicyAgent action policy

STAR Story

Situation

Enterprise data and AI platforms generate fragmented alerts across pipelines, data quality, RAG, ML models, semantic metrics, and AI governance.

Task

Build an autonomous runtime that converts signals into root-cause predictions, governed remediation recommendations, recovery confidence scores, and briefings.

Action

Created synthetic platform signals, historical incident memory, failure patterns, incident triage, blast-radius analysis, root-cause prediction, remediation recommendations, governance policies, autonomous operator simulations, API endpoints, dashboards, tests, Docker, and CI/CD.

Result

Produced a reproducible flagship portfolio project demonstrating autonomous data platform operations and systems-level AI infrastructure thinking.

Project Status

V0.1: Working baseline.