Skip to content

test: audit all test suites for flakiness (esp. -race backend) #365

Description

@krisarmstrong

Problem

CI flakiness erodes trust in the gate and blocks PR auto-merge (forces manual re-runs). Audit stem's test suites to surface and fix/quarantine non-deterministic tests.

Concrete trigger

The Backend (race detector) job flaked twice during the UI token-consolidation work — PR #362 (failed, passed on re-run) and PR #364 (failed, re-ran). UI-only changes can't affect Go tests, and the Frontend (TypeScript) job passed both times, so the failures are non-deterministic. Logs were full of:

ERROR Test execution failed testType=throughput module=benchmark error="test already running"
API error status=409 code=CONFLICT message="A test is already running"

This suggests parallel backend tests share a global test-execution ("running") singleton (testmaster/benchmark) that isn't isolated/reset between cases under -race.

Scope

  • Go: go test -race -count=20 ./... and per-package on the test-execution/benchmark/reflector modules. Look for shared mutable singletons not reset via t.Cleanup, missing locks, or tests that must serialize (-p 1).
  • Frontend unit (vitest): run repeatedly with retries off to find order/timing-dependent specs.
  • E2E (Playwright): run with --retries=0 to expose flakes that retries currently mask.

Acceptance

  • Documented list of flaky tests + root cause.
  • Each fixed (preferred) or quarantined with a tracking note.
  • -race suite green across N consecutive CI runs.

(Filed as part of a trio-wide flakiness audit — see the matching issues in seed and niac.)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions