feat/click velocity detector

README.md

@@ -2,9 +2,9 @@
 
 [![CI](https://github.com/mike-k-git/clickguard/actions/workflows/ci.yml/badge.svg)](https://github.com/mike-k-git/clickguard/actions/workflows/ci.yml)
 
-Clickguard surfaces bot traffic and click fraud signals in web and ad-server access logs. It reads Common Log Format (CLF) input, runs a set of detectors over the parsed events, and scores each actor by the rules they triggered. The result is a ranked list of suspect IPs with band assignments (clear / suspect / fraud) and rule breakdowns.
+Clickguard surfaces bot traffic and click fraud signals in web and ad-server access logs. It reads Common Log Format (CLF) input, runs a set of detectors over the parsed events, and scores each actor by the rules they triggered. The result is a ranked list of suspect IPs with band assignments (clear/suspect/fraud) and rule breakdowns.
 
-It is a portfolio project with a real fraud-mitigation domain. The scoring model is intentionally simple at this stage — the goal is behavioral signal surfacing, not a production-grade verdict engine.
+It is a portfolio project with a real fraud-mitigation domain. The scoring model is intentionally simple at this stage. The goal is behavioral signal surfacing, not a production-grade verdict engine.
 
 ## Build
 
@@ -33,9 +33,9 @@ cat access.log | ./clickguard
 
 ### Options
 
-`--format text|json` — output format. Defaults to `text`.
+`--format text|json` output format. Defaults to `text`.
 
-`--fail-on suspect|fraud` — exit with code 2 if any actor bands at or above the given level. `suspect` is triggered by suspect and fraud; `fraud` by fraud only. Without this flag the exit code reflects only whether the run succeeded.
+`--fail-on suspect|fraud` exit with code 2 if any actor bands at or above the given level. `suspect` is triggered by suspect and fraud; `fraud` by fraud only. Without this flag the exit code reflects only whether the run succeeded.
 
 ### Exit codes
 
@@ -54,14 +54,15 @@ Code 2 is separate from code 1 so a CI step can distinguish a clean log from a b
 Tab-separated table, sorted fraud → suspect → clear, then by score descending within band.
 
 ```
-actor           events  band    score   summary worst
-ip:127.0.0.1    303     fraud   4       low: 4  :high_frequency_ip (300)
-ip:10.0.0.10    4       suspect 2       low: 3  :automation_tool (2)
-ip:10.0.0.1     1       clear   1       low: 1  :automation_tool (1)
-ip:10.0.0.2     1       clear   1       low: 1  :automation_tool (1)
-ip:192.168.1.1  1       clear   1       low: 1  :spam_referer (1)
-ip:192.168.1.10 2       clear   1       low: 2  :spam_referer (1)
-ip:192.168.1.2  1       clear   1       low: 1  :spam_referer (1)
+actor                                     events  band    score   summary worst
+session:172.16.0.1|high-velocity-browser  20      fraud   16      high: 1 :click_velocity (20)
+ip:127.0.0.1                              303     fraud   4       low: 4  :high_frequency_ip (300)
+ip:10.0.0.10                              4       suspect 2       low: 3  :automation_tool (2)
+ip:10.0.0.1                               1       clear   1       low: 1  :automation_tool (1)
+ip:10.0.0.2                               1       clear   1       low: 1  :automation_tool (1)
+ip:192.168.1.1                            1       clear   1       low: 1  :spam_referer (1)
+ip:192.168.1.10                           2       clear   1       low: 2  :spam_referer (1)
+ip:192.168.1.2                            1       clear   1       low: 1  :spam_referer (1)
 ```
 
 ### JSON
@@ -106,14 +107,16 @@ ip:192.168.1.2  1       clear   1       low: 1  :spam_referer (1)
 
 ## Detectors
 
-Each detector is an independent stage. Findings are per `{IP, rule}` pair. One actor triggering the same rule N times produces one finding, not N.
+Each detector is an independent stage. Findings are per `{actor, rule}` pair. One actor triggering the same rule N times produces one finding, not N.
 
 **FreqIp** flags IPs exceeding 300 requests per 60-second sliding window. Returns the first offending window, not the peak.
 
 **UserAgent** flags requests from known automation tools (`python-requests`, `curl`, `wget`, `go-http-client`, `scrapy`) and headless browsers (`HeadlessChrome`, `PhantomJS`), and nil/blank UAs. Catches lazy bots only. Real-UA spoofing is out of scope.
 
 **Referer** flags nil/blank referers and requests from known referer-spam domains. Host matching is exact (normalized: lowercase, `www.` stripped). The spam domain list is configurable.
 
+**ClickVelocity** flags `{ip, ua}` pairs whose click cadence is robotic. Events are grouped into sessions split on a 30-minute idle gap. A session of 5+ clicks fires on its median inter-click interval: <=2s is `:medium`, <=1s is `:high`; or on 5+ clicks inside one second (`:high`). It is the first detector to emit severities above `:low`. All thresholds are configurable.
+
 ## Scoring
 
 The scorer aggregates findings per actor and assigns a band.

config/config.exs

@@ -4,5 +4,6 @@ config :clickguard,
   detectors: [
     Clickguard.Detector.FreqIp,
     Clickguard.Detector.UserAgent,
-    Clickguard.Detector.Referer
+    Clickguard.Detector.Referer,
+    Clickguard.Detector.ClickVelocity
   ]

lib/clickguard.ex

@@ -49,9 +49,13 @@ defmodule Clickguard do
     {:ok, findings}
   end
 
-  @spec actor_totals([Event.t()]) :: %{String.t() => non_neg_integer()}
+  @spec actor_totals([Event.t()]) :: %{{atom(), String.t()} => non_neg_integer()}
   def actor_totals(events) do
-    Enum.frequencies_by(events, fn event -> Event.format_ip(event.ip) end)
+    Enum.reduce(events, %{}, fn event, acc ->
+      acc
+      |> Map.update({:ip, Event.format_ip(event.ip)}, 1, &(&1 + 1))
+      |> Map.update({:session, Event.session_key(event)}, 1, &(&1 + 1))
+    end)
   end
 
   @doc """

lib/clickguard/detector/click_velocity.ex

@@ -0,0 +1,201 @@
+defmodule Clickguard.Detector.ClickVelocity do
+  @moduledoc """
+  Flags sessions that contain suspicious velocity patterns
+
+  Rules (Subject = {IP, UA}):
+    * :click_velocity - median interval within a session is below a threshold
+                        and/or there is a event burst inside a second interval
+
+  All `opts` are overridable. The minimum `min_session_clicks` is 2.
+
+  `high_median_ms` and `burst_threshold` lead to :high severity; `medium_median_ms`
+  to :medium.
+
+  Severity: medium or high
+  """
+  alias Clickguard.{Event, Finding}
+  @behaviour Clickguard.Detector
+
+  @session_gap_ms 1_800_000
+  @min_session_clicks 5
+  @medium_median_ms 2_000
+  @high_median_ms 1_000
+  @burst_threshold 5
+
+  @type evidence :: %{
+          event_count: pos_integer(),
+          sessions: [
+            %{
+              clicks: pos_integer(),
+              median_delta_ms: float(),
+              max_burst: pos_integer(),
+              started_at: DateTime.t(),
+              ended_at: DateTime.t()
+            }
+          ],
+          median_delta_ms: float(),
+          max_burst: pos_integer(),
+          thresholds: map()
+        }
+
+  @impl true
+  def name, do: :click_velocity
+
+  @impl true
+  def detect(events, opts) do
+    detected_at = DateTime.now!("Etc/UTC")
+
+    thresholds = %{
+      session_gap_ms: Keyword.get(opts, :session_gap_ms, @session_gap_ms),
+      min_session_clicks: max(Keyword.get(opts, :min_session_clicks, @min_session_clicks), 2),
+      medium_median_ms: Keyword.get(opts, :medium_median_ms, @medium_median_ms),
+      high_median_ms: Keyword.get(opts, :high_median_ms, @high_median_ms),
+      burst_threshold: Keyword.get(opts, :burst_threshold, @burst_threshold)
+    }
+
+    events
+    |> Enum.group_by(&Event.session_key/1)
+    |> Enum.flat_map(fn {key, evs} ->
+      evs
+      |> Enum.sort_by(& &1.timestamp, DateTime)
+      |> sessionize(thresholds)
+      |> classify(key, thresholds, detected_at)
+    end)
+  end
+
+  defp sessionize(events, thresholds) do
+    chunk_fun = fn event, {session, prev_ts} ->
+      if DateTime.after?(
+           event.timestamp,
+           DateTime.add(prev_ts, thresholds.session_gap_ms, :millisecond)
+         ) do
+        # current event is further than @session_gap_ms from previous
+        # return accumulated events, pass current to new acc and its timestamp
+        {:cont, Enum.reverse(session), {[event], event.timestamp}}
+      else
+        # current event within @session_gap_ms from previous
+        # add it to the acc and update timestamp
+        {:cont, {[event | session], event.timestamp}}
+      end
+    end
+
+    after_fun = fn
+      {session, _prev_ts} -> {:cont, Enum.reverse(session), []}
+    end
+
+    [first | rest] = events
+
+    rest |> Enum.chunk_while({[first], first.timestamp}, chunk_fun, after_fun)
+  end
+
+  defp classify(sessions, key, thresholds, detected_at) do
+    offenders =
+      sessions
+      |> Enum.filter(&(length(&1) >= thresholds.min_session_clicks))
+      |> Enum.map(fn clicks ->
+        intervals = click_intervals(clicks)
+        median = median(intervals)
+        max_burst = max_burst(clicks)
+
+        %{clicks: clicks, median: median, max_burst: max_burst}
+      end)
+      |> Enum.filter(fn session ->
+        session.median <= thresholds.medium_median_ms or
+          session.max_burst >= thresholds.burst_threshold
+      end)
+
+    if offenders != [] do
+      {worst, severity} = worst_session(offenders, thresholds)
+
+      [
+        %Finding{
+          rule: :click_velocity,
+          severity: severity,
+          subject: key,
+          actor_type: :session,
+          evidence: build_evidence(offenders, worst, thresholds),
+          sample_events: Event.sample(worst.clicks),
+          detected_at: detected_at
+        }
+      ]
+    else
+      []
+    end
+  end
+
+  defp worst_session(offenders, thresholds) do
+    rank = fn s ->
+      if s.median <= thresholds.high_median_ms or s.max_burst >= thresholds.burst_threshold,
+        do: 2,
+        else: 1
+    end
+
+    worst = Enum.max_by(offenders, &{rank.(&1), -&1.median})
+    severity = if rank.(worst) == 2, do: :high, else: :medium
+
+    {worst, severity}
+  end
+
+  defp max_burst(clicks) do
+    clicks
+    |> Enum.group_by(fn click ->
+      {seconds, _ms} = DateTime.to_gregorian_seconds(click.timestamp)
+      seconds
+    end)
+    |> Enum.map(fn {_seconds, clicks} -> length(clicks) end)
+    |> Enum.max()
+  end
+
+  defp build_evidence(offenders, worst, thresholds) do
+    evidence =
+      offenders
+      |> Enum.reduce(
+        %{
+          event_count: 0,
+          sessions: [],
+          median_delta_ms: worst.median,
+          max_burst: worst.max_burst,
+          thresholds: thresholds
+        },
+        fn %{clicks: clicks, median: median, max_burst: max_burst}, acc ->
+          %{
+            acc
+            | event_count: acc.event_count + length(clicks),
+              sessions: [
+                %{
+                  started_at: hd(clicks).timestamp,
+                  ended_at: List.last(clicks).timestamp,
+                  clicks: length(clicks),
+                  median_delta_ms: median,
+                  max_burst: max_burst
+                }
+                | acc.sessions
+              ]
+          }
+        end
+      )
+
+    %{evidence | sessions: Enum.reverse(evidence.sessions)}
+  end
+
+  defp click_intervals(clicks) do
+    clicks
+    |> Enum.chunk_every(2, 1, :discard)
+    |> Enum.map(fn [a, b] -> DateTime.diff(b.timestamp, a.timestamp, :millisecond) end)
+  end
+
+  defp median(intervals) do
+    mid = div(length(intervals), 2)
+
+    {i1, i2} = Enum.sort(intervals) |> Enum.split(mid)
+
+    if length(i2) > length(i1) do
+      [med | _] = i2
+      med
+    else
+      [m1 | _] = i2
+      [m2 | _] = Enum.reverse(i1)
+      (m1 + m2) / 2
+    end
+  end
+end

lib/clickguard/detector/freq_ip.ex

@@ -42,7 +42,7 @@ defmodule Clickguard.Detector.FreqIp do
   defp offending_window(events, threshold, window_ms) do
     result =
       events
-      |> Enum.sort_by(& &1.timestamp)
+      |> Enum.sort_by(& &1.timestamp, DateTime)
       |> Enum.reduce_while({:queue.new(), 0}, fn event, {window, count} ->
         cutoff = DateTime.add(event.timestamp, -window_ms, :millisecond)
         {pruned, pruned_count} = prune(&DateTime.before?(&1.timestamp, cutoff), window, count)

lib/clickguard/event.ex

@@ -53,6 +53,25 @@ defmodule Clickguard.Event do
   def format_ip(nil), do: nil
   def format_ip(ip), do: ip |> :inet.ntoa() |> to_string()
 
+  @doc """
+  Returns the {ip, ua} key as a string.
+  Separator is `|` so that `IP` and `UA` never collide.
+
+  ## Examples
+
+    iex> alias Clickguard.Event
+    iex> e = %Event{timestamp: ~U[2016-06-24 13:26:08.003Z], ip: {8193, 3512, 34211, 0, 0, 35374, 880, 29492}, 
+    ...>   user_agent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:151.0) Gecko/20100101 Firefox/151.0", 
+    ...>   referer: "https://google.com", method: "GET", path: "/", status: 200, bytes: 0, country: nil, source: nil, raw: ""}
+    iex> Event.session_key(e)
+    "2001:db8:85a3::8a2e:370:7334|Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:151.0) Gecko/20100101 Firefox/151.0"
+  """
+
+  @spec session_key(t()) :: String.t()
+  def session_key(%__MODULE__{} = event) do
+    "#{format_ip(event.ip)}|#{event.user_agent || ""}"
+  end
+
   @spec sample([%__MODULE__{}]) :: [%__MODULE__{}]
   def sample(events) when length(events) < 5, do: events
   def sample(events), do: Enum.take(events, 3) ++ Enum.take(events, -2)