Skip to content

refactor(#237): add silent-failure severity escalation guidance#269

Open
fullsend-ai-coder[bot] wants to merge 1 commit into
mainfrom
agent/237-silent-failure-severity-escalation
Open

refactor(#237): add silent-failure severity escalation guidance#269
fullsend-ai-coder[bot] wants to merge 1 commit into
mainfrom
agent/237-silent-failure-severity-escalation

Conversation

@fullsend-ai-coder

Copy link
Copy Markdown

Add explicit severity escalation rules for silent-failure findings to the correctness sub-agent, standalone code-review skill, and challenger sub-agent. When code silently produces wrong results without any error signal, severity must be medium or higher regardless of how unlikely the trigger seems today.

Changes:

  • correctness.md: new "Silent-failure severity escalation" section
    setting a medium severity floor for silent failures and naming
    the anti-pattern (probability-based severity discounting)
  • challenger.md: carve-out preventing probability-based downgrading
    of silent-failure findings
  • code-review/SKILL.md: matching guidance in the standalone
    correctness dimension

This addresses the systematic bias where the review agent rated silent-failure bugs as [low] using probability reasoning ("X is likely Y") rather than failure-mode reasoning ("silent and total when it triggers"). Evidence: PR fullsend-ai#2947 annotated-tag finding rated [low] across 4+ passes while a human squad rated [HIGH].


Closes #237

Post-script verification

  • Branch is not main/master (agent/237-silent-failure-severity-escalation)
  • Secret scan passed (gitleaks — c776c248ac9a91f10998ca7107e974a6c9d76514..HEAD)
  • Pre-commit hooks passed (authoritative run on runner)
  • Tests ran inside sandbox

Add explicit severity escalation rules for silent-failure findings
to the correctness sub-agent, standalone code-review skill, and
challenger sub-agent. When code silently produces wrong results
without any error signal, severity must be medium or higher
regardless of how unlikely the trigger seems today.

Changes:
- correctness.md: new "Silent-failure severity escalation" section
  setting a medium severity floor for silent failures and naming
  the anti-pattern (probability-based severity discounting)
- challenger.md: carve-out preventing probability-based downgrading
  of silent-failure findings
- code-review/SKILL.md: matching guidance in the standalone
  correctness dimension

This addresses the systematic bias where the review agent rated
silent-failure bugs as [low] using probability reasoning ("X is
likely Y") rather than failure-mode reasoning ("silent and total
when it triggers"). Evidence: PR fullsend-ai#2947 annotated-tag finding rated
[low] across 4+ passes while a human squad rated [HIGH].

Closes #237
@github-actions

Copy link
Copy Markdown

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

See E2E testing guide for details.

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 10, 2026

Copy link
Copy Markdown

🤖 Finished Review · ❌ Failure · Started 12:58 PM UTC · Completed 12:58 PM UTC
Commit: c776c24 · View workflow run →

@guyoron1

Copy link
Copy Markdown
Owner

/fs-review

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 10, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 3:57 PM UTC · Completed 4:06 PM UTC
Commit: d8e3df7 · View workflow run →

@fullsend-ai-review

Copy link
Copy Markdown

Review — Approve

PR: #269 — refactor(#237): add silent-failure severity escalation guidance
Scope: 3 files changed (30+/1−) in internal/scaffold/fullsend-repo/skills/

Summary

This PR adds explicit severity escalation rules for silent-failure findings to three agent instruction files: the correctness sub-agent, the standalone code-review skill, and the challenger sub-agent. The change is well-motivated by documented evidence (issue #237) showing the review agent systematically rated silent-failure bugs as [low] using probability reasoning across 4+ review passes, while a human squad rated the same finding [HIGH] using failure-mode reasoning.

Dimensions reviewed

Dimension Result
Correctness ✅ No issues. New guidance is internally consistent across all three files. The medium or higher floor does not create a ceiling (permits critical). The challenger exception is well-scoped with a clear predicate ("no error signal"). No contradictions with severity anchoring rules.
Security ✅ No secrets, injection patterns, permission changes, or Unicode anomalies.
Intent & coherence ✅ Traces to issue #237. Scope is appropriate — adding guidance to the challenger and standalone code-review skill is necessary for completeness.
Style & conventions ✅ One initial finding about ### vs **bold:** heading format was removed after adversarial challenge — the ### heading is the correct structural choice for a policy/meta-guidance section.
Docs currency ✅ No documentation is made stale by this change.
Cross-repo contracts ⏭ Skipped — no exported interfaces or API surfaces modified.

Labels: PR modifies agent instruction/skill documentation files in the scaffold

@fullsend-ai-review fullsend-ai-review Bot added ready-for-merge All reviewers approved — ready to merge documentation Improvements or additions to documentation labels Jul 10, 2026
@guyoron1

Copy link
Copy Markdown
Owner

/fs-fix

@fullsend-ai-coder

fullsend-ai-coder Bot commented Jul 10, 2026

Copy link
Copy Markdown
Author

🤖 Finished Fix · ✅ Success · Started 5:38 PM UTC · Completed 5:43 PM UTC
Commit: d8e3df7 · View workflow run →

@fullsend-ai-coder

Copy link
Copy Markdown
Author

🔧 Fix agent — iteration 1 (human-triggered)

No fixes needed. The review bot approved PR #269 with zero findings across all dimensions. The human /fs-fix trigger provided no specific instruction. No code changes were made.

Disagreed (1):

  1. no actionable findings to fix: The review bot approved PR refactor(#237): add silent-failure severity escalation guidance #269 with no findings — all six review dimensions (correctness, security, intent & coherence, style & conventions, docs currency, cross-repo contracts) passed clean. The human trigger (/fs-fix) included no specific instruction (HUMAN_INSTRUCTION='none'). There are no code changes to make.

Tests: passed

Updated by fullsend fix agent

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation ready-for-merge All reviewers approved — ready to merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[RTK+Ponytail] Evidence for #1144: annotated-tag silent-failure bug consistently rated [low] while human rated [high]

1 participant