feat(telemetry): implement constraint evaluation — Phase 3#30
Conversation
…stage scoping Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
…ails Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
…, aggregation) Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
… evaluation Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
|
Warning Rate limit exceeded
Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 43 minutes and 20 seconds. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (18)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7b7b1834b6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| constraintName, | ||
| stage, | ||
| message: result.message ?? `Constraint failed: ${constraintName}`, | ||
| severity: 'error', |
There was a problem hiding this comment.
createViolation hard-codes every failure as severity: 'error', so callers cannot emit warning or critical violations even though ConstraintViolation supports them. This causes downstream policy logic to treat all default constraints as hard errors, which breaks severity-based gating when a constraint is intended to be advisory or critical.
Useful? React with 👍 / 👎.
| const existing = groups.get(event.stage); | ||
|
|
||
| if (existing === undefined) { | ||
| groups.set(event.stage, { | ||
| stage: event.stage, |
There was a problem hiding this comment.
Stage events are grouped only by event.stage, which merges repeated executions of the same stage across different phase values into one bucket. When a stage appears again in a later phase, stage-scoped constraints evaluate against combined historical + in-progress events, so per-stage checks can falsely pass/fail because they are no longer scoped to a single completed stage run.
Useful? React with 👍 / 👎.
1 participant
Summary
Implements Telemetry Phase 3 (Constraint Evaluator) — the critical-path blocker for Wave 1 of the unified ROADMAP. Enables constraint evaluation against pipeline events to prevent runaway token spend, infinite retry loops, and stages that exceed duration limits.
3 new capabilities:
ConstraintEvaluatorImpl) — core evaluation engine withaddConstraint,removeConstraint,evaluate. Supports pipeline-scoped and stage-scoped constraints. Evaluates against all events fromStorageBackend.createMaxRetriesConstraint,createTokenBudgetConstraint,createDurationConstraint,createConsecutiveFailuresConstraint,createLintViolationsConstraint+ batchcreateDefaultConstraintsfactory.threshold(),boolean(),aggregation()for declarative constraint construction with all comparison operators.Quality gates: Build ✅, 112 tests passing ✅, Lint ✅ (0 errors), Performance ✅ (p99 < 50ms with 10 constraints × 1000 events).
Test plan
npx nx build telemetrypassesnpx nx test telemetry— 112 tests passing (6 evaluator + 17 defaults + 15 builders + 1 performance + 40 telemetry + 30 storage + 3 types)npx nx lint telemetry— 0 errorsnpx nx affected -t build,test,lint— all 4 affected projects passRequirements covered
F-020, F-021, F-022, F-023, F-030, F-031, F-032, F-034, NF-002