Skip to content

test(ci): add expiring flaky test registry#914

Closed
luoye520ww wants to merge 1 commit into
KunAgent:developfrom
luoye520ww:codex/test-flaky-registry
Closed

test(ci): add expiring flaky test registry#914
luoye520ww wants to merge 1 commit into
KunAgent:developfrom
luoye520ww:codex/test-flaky-registry

Conversation

@luoye520ww

Copy link
Copy Markdown
Collaborator

Problem

Flaky test quarantine can become permanent when a record has no owner, issue, or expiry metadata.

Root cause

There was no bounded, machine-readable registry contract that CI could validate before consuming flaky-test records.

Scope

This PR adds the registry validator and CLI only. It does not change retry, quarantine, or CI execution behavior.

Changes

  • Add versioned, bounded flaky-test registry validation.
  • Require owner, issue reference, first-seen time, and future expiry.
  • Reject duplicate test/platform records, unsupported fields, invalid timestamps, and out-of-range failure rates.
  • Add a CLI that returns distinct invalid-registry and usage/JSON errors.
  • Add focused Node tests for the contract boundaries.

Safety

  • Registry input is bounded to 512 records and bounded string/number fields.
  • Expiry requires an explicit timezone and must remain in the future.
  • The validator is pure and does not modify CI state or test execution.

Typecheck

```text
npm.cmd run typecheck
```

Passed.

Tests

```text
node --test scripts/flaky-test-registry.test.cjs
node scripts/flaky-test-registry.cjs --file
node scripts/flaky-test-registry.cjs --file
npm.cmd run lint
npm.cmd run build
git diff --check
```

Results: 6 registry tests passed; valid fixture exited 0; invalid fixture exited 1; lint, build, and diff checks passed.

Review performed

  • Functional correctness
  • Input bounds and expiry lifecycle
  • Error/CLI exit behavior
  • Scope and compatibility
  • Test quality

Non-goals

CI retry/quarantine integration and daily flaky-test pressure runs are intentionally deferred to follow-up PRs.

Part of the flaky-test registry work.

@luoye520ww

Copy link
Copy Markdown
Collaborator Author

The Linux packaged check is the same pre-existing guest-target timeout covered by #915; the registry code and typecheck passed. I am not duplicating the packaged smoke fix in this unrelated test registry PR. Please rerun after #915 lands.

@luoye520ww

Copy link
Copy Markdown
Collaborator Author

This work is consolidated into #941, which contains the related CI diagnostics and packaged smoke fix as one tested PR. Closing this duplicate to avoid parallel review and PR noise.

@luoye520ww luoye520ww closed this Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant