⚡ Real-Time Crypto Data Engineering Pipeline ⚡

High-Frequency Liquidation Monitor & Analytics Engine

🏗️ System Architecture

This project implements a robust Event-Driven Data Pipeline designed for high-frequency cryptocurrency market analysis. It moves beyond simple scripting to establish a scalable Operational Analytics Engine.

🔄 Data Flow Pipeline

1. Ingestion Layer (Extract)

   • Source: Kraken WebSocket API (Public Feed).
   • Protocol: Secure WebSockets (WSS) over TCP.
   • Tech: asyncio + websockets library for non-blocking I/O.
   • Throughput: Capable of handling 1000+ ticks/second interactions without backpressure.

2. Processing Layer (Transform)

   • Mechanism: In-Memory Rolling Window Aggregation.
   • Data Structure: optimized collections.deque for O(1) appends/pops.
   • Logic: Real-time calculation of VWAP (Volume Weighted Average Price), Volatility Indices, and Liquidation Volume over configurable windows (default: 5-minute sliding window).
   • Latency: Sub-millisecond processing time per tick.

3. Storage Layer (Load)

   • Database: PostgreSQL 15 (TimescaleDB ready).
   • Driver: asyncpg for high-performance async database access.
   • Schema: Optimized for time-series write interactions.

4. Alerting Layer (Action)

   • Destination: Discord Webhooks (JSON Payloads).
   • Logic: Smart cooling mechanisms and severity classification (LOW/MEDIUM/HIGH/CRITICAL) to reduce alert fatigue.

---

🛠️ Engineering Decisions

Component	Technology	Rationale
Concurrency	Asyncio	Single-threaded event loop eliminates context switching overhead found in threading, ideal for I/O bound WebSocket streams.
State Mgmt	Deque	Provides O(1) performance for sliding window operations, essential for keeping latency consistent as window size grows.
Persistence	PostgreSQL	Chosen over NoSQL for ACID compliance on financial data and powerful analytical query capabilities (Window Functions).
Deployment	Docker	Ensures reproducible "Infrastructure as Code" environments across dev, staging, and production.

---

📊 Database Schema

The liquidation_alerts table is designed as an append-only time-series log.

Column	Type	Description
timestamp	TIMESTAMP	UTC event time (Indexed for range queries)
total_volume_usd	DECIMAL	Aggregated volume triggered in the window
vwap	DECIMAL	Volume Weighted Average Price (Fair Value)
price_change_pct	DECIMAL	Price delta % from window enter to exit
alert_severity	VARCHAR	Classification: LOW, MEDIUM, HIGH, CRITICAL
largest_trade_usd	DECIMAL	Max single trade size in the window

---

🚀 Deployment & Configuration

Prerequisites

• Docker Engine 20.10+
• Docker Compose v2.0+

Environment Configuration (.env)

Configure the pipeline behavior dynamically without rebuilding containers.

# 🧠 Processing Logic
WINDOW_SECONDS=300         # 5 Minutes Rolling Window
LIQUIDATION_THRESHOLD_USD=1000000  # $1M Alert Threshold
ALERT_COOLDOWN_SEC=300     # Anti-Spam Cooldown

# 🔌 Connectivity
DISCORD_WEBHOOK_URL_2=https://discord.com/api/webhooks/...
POSTGRES_USER=admin
POSTGRES_DB=monitor_db

Quick Start (Production Mode)

# 1. Build and detach
docker-compose up --build -d

# 2. Check pipeline status (Health Check)
docker ps
# (Status should be 'Up')

# 3. Stream Operational Logs
docker logs -f liquidation_monitor

---

🔮 Future Engineering Roadmap

• Scalability: Introduce Apache Kafka or Redpanda as an intermediate message broker to decouple Ingestion from Processing.
• High Availability: Deploy multiple ingestion workers behind a Redis deduplication layer.
• Analytics: Integrate Grafana for real-time dashboarding directly from PostgreSQL.