Skip to content

fix(server): bound /healthz database probes with a TTL cache - #1971

Draft
serenakeyitan wants to merge 1 commit into
mainfrom
fix/perf-053-healthz-db-probe-cache
Draft

fix(server): bound /healthz database probes with a TTL cache#1971
serenakeyitan wants to merge 1 commit into
mainfrom
fix/perf-053-healthz-db-probe-cache

Conversation

@serenakeyitan

Copy link
Copy Markdown
Contributor

Summary

Fixes #1716 ([PERF-053][Medium] Public health checks query PostgreSQL without a limit).

/healthz explicitly disables rate limiting and executed SELECT 1 on every request, so public traffic or overly aggressive probes translated into unlimited PostgreSQL round trips — including while the database was already unhealthy.

This PR bounds the database work inside the healthz plugin (packages/server/src/api/healthz.ts):

  • TTL cache — the probe result (healthy or unhealthy) is cached for a short TTL (default 5s), so database load is capped at ~1 probe per window (~0.2 QPS) regardless of request volume. Caching failures too means a down database is also protected from probe hammering.
  • Single-flight — concurrent requests share one in-flight probe, so a cache-expiry burst cannot fan out into parallel SELECT 1s.
  • Contract unchanged — still 200 {"status":"ok"} / 503 {"status":"error","message":"database unreachable"}; rateLimit: false is deliberately retained so orchestrator probes can never be rejected with 429 (a 429 on a health probe can cause false-negative restarts).
  • Testability — TTL is a plugin option (probeCacheTtlMs; 0 disables caching and restores per-request probing).

Design notes — alternatives considered

  1. Pure liveness split (drop the DB check from /healthz): changes the semantics of the Docker HEALTHCHECK (Dockerfile probes /healthz) and every downstream orchestrator config, plus the release QA case contract — a much larger blast radius than the audit requires. The issue's recommended direction explicitly allows "an independent cheap limit/cache" instead.
  2. Route-level rate limit on /healthz: keyed per-IP, so distributed public traffic still multiplies DB hits, and legitimate probes could receive 429 and trigger false restarts.
  3. Ingress restriction: deployment-platform-specific (Railway/Fly/K8s/Docker); cannot be solved for all users from this repo.

The Docker HEALTHCHECK cadence (30s interval, 3 retries) is far coarser than the 5s TTL, so orchestrator failure-detection behavior is effectively unchanged (worst-case staleness ≤ 5s).

Verification

  • pnpm check — pass
  • pnpm typecheck — pass (10/10 tasks)
  • pnpm --filter @first-tree/server test — 238 files / 2541 tests, all pass
  • New packages/server/src/__tests__/healthz-probe-cache.test.ts covers: rate-limit config retained; repeated requests within TTL hit the DB once; re-probe after TTL expiry; unhealthy result cached (down DB not hammered); recovery reported on first probe after expiry; concurrent requests share one in-flight probe; probeCacheTtlMs: 0 probes every request.
  • api-small-routes-extra.test.ts updated to pass probeCacheTtlMs: 0 where a single registration exercises both branches.

QA note

This touches the boot/health path. The existing QA case packages/qa/cases/cross-surface/release-boot-health.md already covers /healthz returning {"status":"ok"} from the production image; the endpoint contract is unchanged, so the case stays valid. Flagging that a formal run of that case is warranted before release.

/healthz disables rate limiting and previously executed SELECT 1 on every
request, so public traffic or aggressive probes translated into unlimited
database round trips — including while the database was already unhealthy.

Cache the probe result (healthy or not) for a short TTL and share a single
in-flight probe across concurrent requests, bounding database load to at
most one probe per window regardless of request volume. The endpoint
contract (200/503 bodies) and rateLimit: false are unchanged so
orchestrator probes are never rejected.

Fixes #1716
@serenakeyitan serenakeyitan added fire_wip GoF: draft PR in progress fire_submitted GoF: PR submitted for maintainer review (stays draft) and removed fire_wip GoF: draft PR in progress labels Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fire_submitted GoF: PR submitted for maintainer review (stays draft)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[PERF-053][Medium] Public health checks query PostgreSQL without a limit

1 participant