feat(config): implement runtime configuration drift auditing

[ekwe7]

closes #85

Implement Runtime Configuration Auditing and Drift Detection

Summary

This PR introduces runtime configuration auditing to detect and surface production configuration drift before it results in silent outages or unexpected behavior.

The implementation periodically snapshots runtime configuration, compares it against the repository-defined baseline, classifies detected drift, and routes critical deployment-scoped alerts through PagerDuty. A historical dashboard is also added to provide visibility into configuration changes over time.

Problem

During incident response, emergency fixes, or manual operational changes, production configuration can diverge from the expected repository-defined state.

Without automated auditing:

• Configuration drift can go unnoticed
• Production behavior may differ from tested deployments
• Incident root-cause analysis becomes more difficult
• Silent outages can occur due to unintended configuration changes
• Teams lack visibility into historical configuration changes

Solution

This PR introduces a runtime configuration auditing pipeline that continuously validates deployed configuration against a known baseline.

Key Capabilities

• Periodic runtime configuration snapshots
• Baseline comparison against committed repository configuration
• Drift detection and classification
• PagerDuty alerting for critical deployment-scoped drift
• Historical drift visualization and analysis

Implementation

Configuration Snapshot Collector

Added a collector responsible for:

• Capturing runtime configuration state
• Running on a 5-minute interval
• Normalizing configuration data for comparison
• Persisting snapshot history

Drift Detection Engine

Implemented a comparator that evaluates runtime snapshots against the repository baseline.

Supported drift categories:

Value Changes

DATABASE_URL
old: postgres://primary
new: postgres://replica

Added Keys

NEW_FEATURE_FLAG=true

Removed Keys

PAYMENT_TIMEOUT_MS

Alert Routing

Added drift classification and alert routing logic.

Critical deployment-scoped drift events:

• Generate PagerDuty alerts
• Include affected deployment details
• Include drift type and impacted configuration keys
• Support operational triage workflows

Drift Dashboard

Implemented a dashboard providing:

• Drift history timeline
• Configuration change summaries
• Drift classification breakdowns
• Deployment-specific filtering
• Historical auditing support

Technical Bounds

Snapshot Interval

5 minutes

Baseline Source

Committed configuration files in the repository

Supported Drift Types

• Value changes
• Added keys
• Removed keys

Alert Destination

PagerDuty

for deployment-scoped critical drift events.

Operational Benefits

Reliability

• Detects unexpected configuration changes quickly
• Reduces risk of silent outages
• Improves deployment consistency

Incident Response

• Provides immediate visibility into configuration drift
• Accelerates root-cause identification
• Supports rollback and remediation workflows

Compliance and Auditing

• Maintains historical configuration records
• Tracks configuration evolution over time
• Supports operational audit requirements

Validation

Performed validation for:

• Snapshot generation
• Baseline comparison accuracy
• Added key detection
• Removed key detection
• Value change detection
• PagerDuty alert routing
• Dashboard history rendering
• Drift classification accuracy

Acceptance Criteria

• Runtime configuration snapshots collected every 5 minutes
• Runtime configuration compared against repository baseline
• Value changes detected
• Added keys detected
• Removed keys detected
• Critical deployment-scoped drift routed to PagerDuty
• Drift history dashboard implemented
• Historical drift records retained for auditing

[JamesEjembi]

@ekwe7 CI Failed