Problem
CI flakiness erodes trust in the gate and blocks PR auto-merge (forces manual re-runs). Audit stem's test suites to surface and fix/quarantine non-deterministic tests.
Concrete trigger
The Backend (race detector) job flaked twice during the UI token-consolidation work — PR #362 (failed, passed on re-run) and PR #364 (failed, re-ran). UI-only changes can't affect Go tests, and the Frontend (TypeScript) job passed both times, so the failures are non-deterministic. Logs were full of:
ERROR Test execution failed testType=throughput module=benchmark error="test already running"
API error status=409 code=CONFLICT message="A test is already running"
This suggests parallel backend tests share a global test-execution ("running") singleton (testmaster/benchmark) that isn't isolated/reset between cases under -race.
Scope
- Go:
go test -race -count=20 ./... and per-package on the test-execution/benchmark/reflector modules. Look for shared mutable singletons not reset via t.Cleanup, missing locks, or tests that must serialize (-p 1).
- Frontend unit (vitest): run repeatedly with retries off to find order/timing-dependent specs.
- E2E (Playwright): run with
--retries=0 to expose flakes that retries currently mask.
Acceptance
- Documented list of flaky tests + root cause.
- Each fixed (preferred) or quarantined with a tracking note.
-race suite green across N consecutive CI runs.
(Filed as part of a trio-wide flakiness audit — see the matching issues in seed and niac.)
Problem
CI flakiness erodes trust in the gate and blocks PR auto-merge (forces manual re-runs). Audit stem's test suites to surface and fix/quarantine non-deterministic tests.
Concrete trigger
The
Backend (race detector)job flaked twice during the UI token-consolidation work — PR #362 (failed, passed on re-run) and PR #364 (failed, re-ran). UI-only changes can't affect Go tests, and theFrontend (TypeScript)job passed both times, so the failures are non-deterministic. Logs were full of:This suggests parallel backend tests share a global test-execution ("running") singleton (testmaster/benchmark) that isn't isolated/reset between cases under
-race.Scope
go test -race -count=20 ./...and per-package on the test-execution/benchmark/reflector modules. Look for shared mutable singletons not reset viat.Cleanup, missing locks, or tests that must serialize (-p 1).--retries=0to expose flakes that retries currently mask.Acceptance
-racesuite green across N consecutive CI runs.(Filed as part of a trio-wide flakiness audit — see the matching issues in seed and niac.)