Sink Health Monitoring: Anomaly Detection Engine

[joejohnson123]

Overview

Implement anomaly detection for sink monitors using z-score statistical analysis on a rolling window of metric samples.

Depends on: #1 (Schema — metric_samples table), #2 (Context Module), #3 (Evaluation Worker)

Module: Sequin.Monitors.AnomalyDetector

Core Algorithm: Z-Score

The z-score measures how many standard deviations a value is from the mean:

z = (value - mean) / stddev

If z > sensitivity_threshold, the value is anomalous.

Sensitivity Thresholds

Sensitivity	Z-Score Threshold	Meaning
:low	3.0	Only extreme outliers (99.7th percentile)
:medium	2.0	Notable deviations (95.4th percentile)
:high	1.5	Sensitive to smaller changes (86.6th percentile)

API

@spec analyze(monitor_id :: String.t(), current_value :: integer(), sensitivity :: atom()) ::
  {:normal, z_score :: float()} | {:anomaly, z_score :: float()}
def analyze(monitor_id, current_value, sensitivity)

Sample Collection

• The evaluation worker records a sample every 30 seconds (each evaluation cycle)
• Stored in sink_monitor_metric_samples table
• Rolling window: 1 hour (120 data points at 30s intervals)
• Minimum samples required before detection activates: 30 (~15 minutes of data)
  • Before minimum is reached, return {:insufficient_data, sample_count}

Edge Cases

• Zero stddev (all values identical): Treat any deviation as anomaly if value differs from mean
• All zeros: If mean is 0 and current value > 0, flag as anomaly (something started happening)
• Gradual drift: Z-score naturally adapts as the rolling window moves — a gradual increase won't trigger but a sudden spike will
• New monitor: First 15 minutes are warm-up period, no alerts fired

Pruning

• prune_old_samples/0 — delete samples older than 2 hours
• Called by the evaluation worker after each cycle
• Use DELETE FROM sink_monitor_metric_samples WHERE sampled_at < $1

Integration with Evaluation Worker

Extend EvaluateMonitorsWorker to handle anomaly monitors:

1. Record metric sample
2. Call AnomalyDetector.analyze/3
3. If {:anomaly, z_score}:
   • Check cooldown
   • Fire alert with message including z-score and expected range
4. If {:normal, _} and active alert exists:
   • Resolve alert
5. Call prune_old_samples/0 periodically (every 10th cycle)

Alert Message Format

• Triggered: "Anomaly detected: {metric} is {value} (z-score: {z:.1f}, expected range: {mean-threshold*stddev:.0f}–{mean+threshold*stddev:.0f}) on sink {sink_name}"
• Resolved: "Anomaly resolved: {metric} returned to normal ({value}) on sink {sink_name}"

Files to Create

• lib/sequin/monitors/anomaly_detector.ex
• test/sequin/monitors/anomaly_detector_test.exs

Files to Modify

• lib/sequin/monitors/evaluate_monitors_worker.ex — add anomaly monitor evaluation branch

Implementation Notes

• Pure Elixir math — no external stats libraries needed
• mean = Enum.sum(values) / length(values)
• stddev = :math.sqrt(Enum.sum(Enum.map(values, fn v -> (v - mean) ** 2 end)) / length(values))
• Keep it simple for v1 — z-score is well-understood and explainable to users
• Future improvement: exponential weighted moving average (EWMA) for better recency bias

Acceptance Criteria

• Correctly identifies statistical anomalies at all 3 sensitivity levels
• Returns :insufficient_data when not enough samples collected
• Handles edge cases (zero stddev, all zeros, single value)
• Samples are recorded and pruned correctly
• Alert messages include z-score and expected range
• Warm-up period prevents false positives on new monitors
• Tests cover normal distributions, spike detection, and edge cases