Senior Unified Stack: Monitoring + Shared Infrastructure (100% Hardened)

This repository contains the Docker infrastructure for a unified application stack that powers Postly, Learnest, JobSpy Scraper, and a full observability platform — all on one VPS. Everything is 100% hardened (Fort Knox grade), resource-capped, and production-ready.

---

🏗️ What's Inside

📊 Monitoring & Observability (Auto-Configured)

Service	Purpose	Memory Limit
Grafana	Dashboards + Alerting → Discord	96M
VictoriaMetrics	Time-series metrics storage	160M
VictoriaLogs	High-performance log storage	160M
Vector	Log/metric collection agent	96M
docker-socket-proxy	Secure Docker API gatekeeper	32M

3 pre-loaded dashboards: System Overview, Docker Containers, Docker Logs Explorer
7 auto-provisioned alerts: High CPU/Memory/Disk, Container Down, etc. → Discord webhook

🗄️ Shared Infrastructure (All Apps Connect Here)

Service	Purpose	Memory Limit	Shared By
PostgreSQL (pgvector/pg17)	Relational DB + vector search	512M	Monitoring + Postly + Learnest
Redis 7	Caching + queues + rate limiting	192M	Monitoring + Postly + Learnest
RabbitMQ	Message broker	128M	Learnest (optional)

One database server, one Redis, one network. Every app gets its own database/user/Redis DB index. No duplicate infrastructure wasting RAM.

☁️ Media Storage (Cloudinary)

Postly uses Cloudinary for image and media storage — a cloud-based image/video management platform:

Feature	Cloudinary
Free tier	25 GB storage, 25 GB bandwidth/mo
Image transformations	✅ Built-in CDN + on-the-fly transforms
Setup	5 min (create account + get API keys)

---

⚡ Resource Footprint (4GB VPS)

Layer	Components	Memory Limit	Typical Idle
Monitoring	Grafana, VictoriaMetrics, VictoriaLogs, Vector, Socket Proxy	544M	~200-300M
Shared Infra	PostgreSQL, Redis, RabbitMQ (optional)	832M	~400-500M
Postly apps	API, Scraper, Bot, Web	704M	~300-400M
Learnest apps	Server, Web, Student Web, Admin	416M	~200-300M
JobSpy Scraper	Python job scraper	384M	~200-300M
Total	~18 containers	~2.9 GB	~1.3-1.8 GB

✅ Easily fits within 4GB with ~1GB headroom for OS and spikes. All containers have strict memory: limits — no single service can OOM the host.

---

✨ Key Features

• ✅ Shared Everything — One PostgreSQL, one Redis, one network. Apps talk to each other by container name.
• ✅ Media Storage — Cloudinary for Postly image/media management. No local storage overhead.
• ✅ Actually Monitors — 3 dashboards, 7 alert rules, live log explorer, Discord alerts.
• ✅ 100% Hardened — Zero-capability containers, read-only filesystems, Docker Socket Proxy.
• ✅ Auto-Init — PostgreSQL auto-creates databases + roles for every app on first boot.
• ✅ Strict Memory Capping — Every service has a limit. No OOM kills on your VPS.
• ✅ Production Auth — SCRAM-SHA-256 for Postgres, password-protected Redis.

---

🚀 Quick Start

1. Setup Environment

# Edit .env — set strong passwords for EVERYTHING
# Add your Cloudinary credentials for Postly media storage
nano .env

2. Start the Stack

docker-compose up -d

3. Verify Everything

# All containers should show 'healthy'
docker ps

# PostgreSQL should have all databases
docker exec postgres psql -U postgres -c "\l"
# Expected: postgres, postly (and learnest if configured in POSTGRES_MULTIPLE_DATABASES)

# Test Redis
docker exec redis redis-cli -a "${REDIS_PASSWORD}" PING
# Expected: PONG

4. Access Grafana

Open http://localhost:3001 — login: admin / your GF_SECURITY_ADMIN_PASSWORD

See 3 dashboards in "Senior Monitoring" folder with live data from all containers.

---

🛡️ Services Overview

Databases

Service	Container Name	Internal Port	Host Port	Auth
PostgreSQL	postgres	5432	5433 (localhost)	SCRAM-SHA-256
Redis	redis	6379	6380 (localhost)	Password
RabbitMQ	rabbitmq	5672	— (internal)	Password

Monitoring

Service	Container Name	Internal Port	Host Port	Purpose
Grafana	grafana	3000	3001 (localhost)	Dashboards & Alerting
VictoriaMetrics	victoriametrics	8428	—	Metrics storage
VictoriaLogs	victorialogs	9428	—	Log storage
Vector	vector	—	—	Data pipeline agent
docker-socket-proxy	docker-socket-proxy	2375	—	Docker API gatekeeper

---

🔌 Connecting Your Apps

PostgreSQL — Per-App Databases

App	Database	Role	Redis DB Index
Monitoring	postgres	postgres	0
Postly	postly	postly	1
Learnest	learnest	learnest	2

Postly connection:

DATABASE_URL=postgresql://postly:${POSTLY_DB_PASSWORD}@postgres:5432/postly
REDIS_URL=redis://:${REDIS_PASSWORD}@redis:6379/1

Learnest connection:

DATABASE_URL=postgresql://learnest:${LEARNEST_DB_PASSWORD}@postgres:5432/learnest
REDIS_ADDR=redis:6379
REDIS_DB=2

Media Storage — Cloudinary

# Postly uses Cloudinary for media uploads
CLOUDINARY_URL=cloudinary://${CLOUDINARY_API_KEY}:${CLOUDINARY_API_SECRET}@${CLOUDINARY_CLOUD_NAME}

---

📊 Data Flow

Your VPS
  │
  ├─ Containers (stdout/stderr)
  │   └──→ Vector ──→ VictoriaLogs ──→ Grafana (Logs Explorer)
  │
  ├─ Host metrics (/proc, /sys)
  │   └──→ Vector ──→ VictoriaMetrics ──→ Grafana (System Overview)
  │
  ├─ Docker container stats
  │   └──→ Vector ──→ VictoriaMetrics ──→ Grafana (Docker Containers)
  │                                     └──→ Alerts → Discord
  │
  ├─ Application metrics (/metrics endpoints)
  │   └──→ VictoriaMetrics ──→ Grafana (App-specific dashboards)
  │
  └─ App data (Postly + Learnest)
      └──→ PostgreSQL (shared) + Redis (shared) + Cloudinary (media storage)

---

🧰 Maintenance

# Check all container health
docker ps

# View live resource usage
docker stats

# View logs for a specific service
docker-compose logs -f grafana

# Backup all databases
./scripts/backup.sh

# Add a new database to PostgreSQL
./scripts/add_db.sh new_database_name

Backup Strategy

• Daily: pg_dump of all app databases → /backups (7-day retention)
• Redis: SAVE + copy RDB file daily
• Media Storage: Cloudinary manages durability — no local backup needed

---

🔒 Security Hardening

Measure	Applied To
read_only: true (immutable filesystem)	All containers
cap_drop: [ALL] (no kernel capabilities)	All containers
no-new-privileges:true	All containers
tmpfs: /tmp (ephemeral temp storage)	All containers
No 0.0.0.0 binding except reverse proxy	All services (localhost-only)
Docker Socket Proxy (no direct socket access)	All app containers
SCRAM-SHA-256 password encryption	PostgreSQL
Dangerous Redis commands renamed	FLUSHALL, FLUSHDB, DEBUG disabled

---

📈 Scaling

Resource	Current	If You Upgrade VPS
RAM	~3 GB allocated	Increase PG to 1GB, Redis to 256M, VM to 256M
PostgreSQL	512M / 100 connections	Increase shared_buffers and max_connections
Storage	Depends on retention	Add --retentionPeriod=30d to VictoriaMetrics
Apps	2 + 1 scraper	Add more services to the same network