fix(server): bound /healthz database probes with a TTL cache - #1971
Draft
serenakeyitan wants to merge 1 commit into
Draft
fix(server): bound /healthz database probes with a TTL cache#1971serenakeyitan wants to merge 1 commit into
serenakeyitan wants to merge 1 commit into
Conversation
/healthz disables rate limiting and previously executed SELECT 1 on every request, so public traffic or aggressive probes translated into unlimited database round trips — including while the database was already unhealthy. Cache the probe result (healthy or not) for a short TTL and share a single in-flight probe across concurrent requests, bounding database load to at most one probe per window regardless of request volume. The endpoint contract (200/503 bodies) and rateLimit: false are unchanged so orchestrator probes are never rejected. Fixes #1716
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #1716 ([PERF-053][Medium] Public health checks query PostgreSQL without a limit).
/healthzexplicitly disables rate limiting and executedSELECT 1on every request, so public traffic or overly aggressive probes translated into unlimited PostgreSQL round trips — including while the database was already unhealthy.This PR bounds the database work inside the healthz plugin (
packages/server/src/api/healthz.ts):SELECT 1s.200 {"status":"ok"}/503 {"status":"error","message":"database unreachable"};rateLimit: falseis deliberately retained so orchestrator probes can never be rejected with 429 (a 429 on a health probe can cause false-negative restarts).probeCacheTtlMs;0disables caching and restores per-request probing).Design notes — alternatives considered
/healthz): changes the semantics of the DockerHEALTHCHECK(Dockerfile probes/healthz) and every downstream orchestrator config, plus the release QA case contract — a much larger blast radius than the audit requires. The issue's recommended direction explicitly allows "an independent cheap limit/cache" instead./healthz: keyed per-IP, so distributed public traffic still multiplies DB hits, and legitimate probes could receive 429 and trigger false restarts.The Docker
HEALTHCHECKcadence (30s interval, 3 retries) is far coarser than the 5s TTL, so orchestrator failure-detection behavior is effectively unchanged (worst-case staleness ≤ 5s).Verification
pnpm check— passpnpm typecheck— pass (10/10 tasks)pnpm --filter @first-tree/server test— 238 files / 2541 tests, all passpackages/server/src/__tests__/healthz-probe-cache.test.tscovers: rate-limit config retained; repeated requests within TTL hit the DB once; re-probe after TTL expiry; unhealthy result cached (down DB not hammered); recovery reported on first probe after expiry; concurrent requests share one in-flight probe;probeCacheTtlMs: 0probes every request.api-small-routes-extra.test.tsupdated to passprobeCacheTtlMs: 0where a single registration exercises both branches.QA note
This touches the boot/health path. The existing QA case
packages/qa/cases/cross-surface/release-boot-health.mdalready covers/healthzreturning{"status":"ok"}from the production image; the endpoint contract is unchanged, so the case stays valid. Flagging that a formal run of that case is warranted before release.