Skip to content

evaluation.md

Latest commit

 

History

History
67 lines (50 loc) · 2.25 KB

evaluation.md

File metadata and controls

67 lines (50 loc) · 2.25 KB
Raw

Evaluation Design

Back to README

AgentCode does not evaluate whether a user can handwrite a specific piece of code. It evaluates whether the user can deliver a shippable engineering result and correctly review code generated by AI.

Task Mode Evaluation

Task Mode is mainly based on deterministic checks:

  • Patch applies cleanly.
  • Project installs successfully.
  • Lint / typecheck passes when configured.
  • Existing tests pass.
  • Public tests pass.
  • Hidden tests pass.
  • Challenge-specific behavior checks pass.
  • Hardcoded outputs or test-bypassing shortcuts are rejected when needed.

The result should be transparent:

  • accepted: required checks pass.
  • failed: tests or validation failed.
  • needs_review: automated checks pass, but the challenge needs manual or rubric-based inspection.

LLM-based inspection can be added as an auxiliary feedback layer for suspicious patches, hardcoded fixes, or maintainability concerns. It should support deterministic checks, not replace them.

Review Mode Evaluation

Review Mode uses structured rubric scoring:

  • Correct merge decision.
  • Required issues found.
  • Correct severity.
  • Correct affected area.
  • Quality of explanation.
  • Penalty for false positives.
  • Penalty for missing the central risk.

For V0, the rubric is the source of truth. LLM scoring can help normalize free-form answers later, but it should not be the only authority.

Review Rubric Example

expectedFindings:
  - id: missing-permission-check
    severity: critical
    category: security
    required: true
    evidence:
      - "The new endpoint updates shared resources without checking ownership."
    acceptableAnswers:
      - "missing authorization"
      - "no permission check"
      - "user can modify another user's resource"
mergeDecision: request_changes
falsePositivePenalty: medium

Evaluation Principles

  • Deterministic checks come first.
  • Feedback must be readable and actionable.
  • Review Mode answers do not need to match the reference answer word for word, but they must identify the central risks.
  • Automated evaluation should not encourage users to write code only for tests.
  • Evaluation should help users understand whether they are missing implementation skill, testing skill, or code judgment.