This project simulates a data fabric architecture for the Integrated National Security command and control System
It demonstrates how heterogeneous security data streams can be
- Ingested
- Standardized
- Published to an event backbone
- Visualized through a common Operating Picture
The system employs an event-driven microservices architecture using apache kafka as the central event streaming platform.
The main focus of this project is to demonstrate the flow of data rather than persistence, AI integration and advanced security controls. These features will be added at a later stage.
Data feeds -> Gateway -> Event Backbone -> Visualization
This service acts as the API boundary of the system.
Responsibilities:
- Validating incoming requests using pydantic
- Publishing structured events to kafka
- Managing Kafka producer lifecycle
- Exposing REST endpoints for event submission
Kafka serves as the event streaming backbone.
Responsibilities
- Asynchronous event distribution
- Decoupling producers and consumers
- Scalable event-driven communication
Kafka runs inside docker using Docker Compose.
The Generator service will:
- Simulate multi-agency data streams
- Produce synthentic security events
- Publish directly or through the gateway
This abstracts data simulation from the api concerns and maintains the SOLID design principles.
Visualization is achieved by a dashboard powered by the streamlit library. The dashboard should:
- Consume kafka events
- Provide a real-time Common Operating Picture
- Demonstrate operational awareness dashboards
This project is conceptually a python-FastAPI project.
- Python 3.12+
- FastAPI
- Apache Kafka
- Docker & Docker Compose
- uv (dependency manager)
- Streamlit (visualization)
Run these after successfully cloning the project.
From the project root:
docker compose up -d
docker psOn the gateway subdirectory
cd gateway
uv run uvicorn app.main:app --reloadThis should open the fastAPI swagger UI documentation
PS: This file is subject to change as I progress with the project.