Jetmon 2 — Site health platform

[chrisbliss18]

Work in progress. This branch (v2) is the ambitious successor to the Go rewrite started in refactor/jetmon2 (PR #60). It includes everything from that branch and extends it with a new architecture and direction.

---

What changed from PR #60

PR #60 scoped Jetmon 2 as a drop-in replacement: same interfaces, same schema, same behaviour — just Go instead of Node.js + C++. That work is complete and forms the base of this branch.

This branch pivots to a larger goal: Jetmon 2 as a full site health monitoring platform, not just an uptime tracker. The key additions:

• Event-sourced architecture. Site state is derived from an event log, not a mutable status column. The event log is the source of truth; the site row carries a denormalized projection for fast reads. Full design in EVENTS.md.
• Five-layer test taxonomy. Reachability → Transport & Security → Infrastructure & Edge → Application Response → Content Integrity, plus Reverse Checks (agent-reported signals from inside WordPress). ~55 v1 items, ~55 v2, ~40 v3. Full taxonomy in TAXONOMY.md.
• Site → Endpoint → Check hierarchy. Sites have multiple endpoints; each endpoint has multiple checks of different types. Site state rolls up from endpoint state, which rolls up from check results. Rollup rules are explicit and configurable per site.
• Multi-state vocabulary. Up, Warning, Degraded, Seems Down, Down, Paused, Maintenance, Unknown. Unknown is not downtime — monitor-side failures never inflate customer downtime figures.
• Competitor-parity public REST API. Five capability groups: status and state, events and history, SLA statistics (uptime %, response time p95/p99, MTTR), monitor management (CRUD, pause, resume, trigger-now), and alert contacts with outbound webhooks. Full design in ROADMAP.md.
• Gradual rollout with back-compat. The existing site_status column keeps receiving derived writes so current consumers are not broken. New capabilities are additive; consumers adopt progressively.

Architectural decisions are locked in AGENTS.md so they are enforced consistently across all changes.

---

Why Go

The current architecture uses forked Node.js processes (8–16MB RSS each at startup, 53MB limit before recycling) as workers, plus a compiled C++ addon to escape Node's event loop for blocking network I/O. Go eliminates both constraints:

• Goroutines start at ~4KB of stack and grow on demand, making 50,000 concurrent checks on a single host practical without the memory overhead of forked processes or libuv thread pools
• net/http and crypto/tls are first-class stdlib packages — no native addon, no node-gyp, no compilation step during deployment
• net/http/httptrace provides DNS, TCP, TLS, and TTFB timing hooks as separate measurements within each check, for free
• Single static binary deployment with no runtime dependencies, no node_modules, and no addon rebuild on Node.js version upgrades
• Built-in profiling via pprof, race detector via go test -race, and a mature testing ecosystem
• Graceful goroutine lifecycle management replaces the fragile worker spawn/recycle/evaporate lifecycle

The Veriflier is rewritten in Go as well, replacing the Qt C++ dependency with a lightweight Go HTTP service. The protocol between Monitor and Verifliers moves from custom HTTPS to gRPC, providing type-safe contracts, built-in retries, and bidirectional streaming for future use.

Benefits of the Rewrite

Memory

The current architecture forks Node.js worker processes that start at 8–16MB RSS and are recycled once they reach 53MB. With a typical deployment of 8–16 workers, the process tree consumes 240–850MB of resident memory just for worker overhead, before any check data is counted.

Jetmon 2 runs as a single process. Go goroutines start at 4KB of stack and grow on demand. A pool of 1,000 concurrent goroutines costs roughly 4MB of stack. Total process RSS for an equivalent workload is estimated at 50–150MB — a 75–90% reduction in memory consumption per host.

Concurrent Checks

Current concurrency is bounded by the number of worker processes. Each worker is a single-threaded Node.js process; practical concurrency per host is in the low hundreds.

Go's goroutine scheduler makes 10,000+ concurrent in-flight checks on a single host practical with no additional configuration. At a conservative network timeout of 10 seconds and average site response time of 200ms, a pool of 1,000 goroutines sustains approximately 5,000 check completions per second — an estimated 10–50× increase in concurrent checks per host.

Throughput

The current architecture crosses a process boundary on every unit of work: master dispatches via IPC, worker receives and processes, replies via IPC, master aggregates. Each crossing involves serialisation, a context switch, and V8 event loop scheduling on both ends.

Jetmon 2 replaces all IPC with Go channel sends, which are in-process and order-of-magnitude cheaper. Estimated throughput improvement: 3–10× more sites checked per second per host under equivalent conditions.

Check Scheduling Accuracy

The current system uses setTimeout and setInterval for round scheduling, subject to V8 event loop delay — a busy loop can delay a callback by tens to hundreds of milliseconds, introducing jitter into RTT measurements.

Go's time.Ticker fires with OS-level timer precision. RTT measurements from net/http/httptrace are taken inside the HTTP stack with no event loop between the measurement point and the timer.

Deployment Speed

Current deployment requires npm install, a node-gyp rebuild of the native C++ addon, and a coordinated process restart. A failed addon compilation blocks deployment entirely.

Jetmon 2 deploys as a single static binary. Deployment is: copy binary, systemctl restart jetmon2. Total deployment time drops from several minutes to under 30 seconds.

Mean Time to Recovery

A worker process crash requires the master to detect the exit, spawn a replacement, and wait for initialisation — several seconds, with in-flight checks unresolved.

In Jetmon 2, a panicking goroutine is recovered by a deferred handler, the result counted as an error, and a replacement goroutine immediately spawned — recovery in the low milliseconds. For a full process crash, systemd restarts the binary; with Go's fast startup, the process is accepting work again in under 2 seconds.

Operational Complexity

The current system requires managing Node.js version compatibility, native addon compilation, npm dependency trees, and the fragile worker spawn/recycle lifecycle.

Jetmon 2 eliminates all of this. One artifact to manage: the Go binary. No node-gyp, no npm, no Node.js version management.

---

Build order

1. Schema — jetmon_endpoints, jetmon_events, updated site row projection, back-compat site_status derived write
2. Probe runner — replaces per-site HTTP check loop; iterates endpoints; owns idempotent event dedup
3. Check types — DNS, TLS cert expiry, redirect chain, keyword, TTFB (v1 taxonomy)
4. Public REST API — status/state, events/history, SLA statistics, monitor management, alert contacts/webhooks

🤖 Generated with Claude Code