Skip to content

SPEC.md

Latest commit

 

History

History
337 lines (284 loc) · 19.5 KB

SPEC.md

File metadata and controls

337 lines (284 loc) · 19.5 KB
Raw

SPEC.md — find-bugs: Universal Repository Bug-Audit Skill

This document is the design specification behind BUILD_GUIDE.md and PROMPT_FOR_AGENT.md. It explains why the skill is built the way it is, so that anyone (or any model) extending or debugging it later understands the intent, not just the mechanics. If BUILD_GUIDE.md and this document ever disagree, BUILD_GUIDE.md is the source of truth for what gets written to disk — this document explains the reasoning behind those choices.

1. Objective

Produce a single skill, find-bugs, that:

  1. Works across Claude Code, OpenCode, Codex CLI, and any other CLI coding agent that follows the open Agent Skills standard (SKILL.md with name
    • description frontmatter), with AGENTS.md as a fallback for tools that support neither.
  2. Triggers automatically on natural phrasing like "find bugs in the codebase", and can also be invoked directly (/find-bugs, $find-bugs, /skills).
  3. Performs a full-repository audit for six categories of bugs: logical, typos, formula/calculation, inconsistencies, runtime, and compile-time/type/syntax errors.
  4. Runs at maximum reasoning depth/effort on every tool that exposes such a control.
  5. Produces a single, categorized BUG_REPORT.md.
  6. Is robust enough to be executed by a less capable model without losing track of a large codebase — this drove several specific design choices below (externalized scratch state, JSONL findings, batch processing, explicit schemas instead of free-form notes).

2. Design principles

Common-denominator format. Anthropic's Agent Skills docs, OpenAI's Codex Skills docs, and third-party compatibility surveys all converge on the same minimum viable SKILL.md: a YAML frontmatter block with name and description, followed by Markdown instructions. Anything beyond that (allowed-tools, context: fork, hooks, effort) is tool-specific but "safely ignored by agents that don't support them" per the cross-tool comparisons. So: write one SKILL.md using only the universal fields plus a small number of high-value, safely-ignorable extras (here, just effort: max for Claude Code's adaptive-effort system), and place copies of the same folder wherever each tool looks.

Progressive disclosure. Keep SKILL.md itself well under the ~5,000-word guidance so the model isn't overwhelmed when the skill loads. The large, mostly-static bug checklist lives in references/bug-taxonomy.md (loaded only once, in Phase 3); the report skeleton lives in assets/report_template.md; the static-analysis logic lives in scripts/run_static_checks.sh. SKILL.md itself is the orchestration layer.

Externalize state, don't trust working memory. A weaker model auditing a 200-file repo will lose track of earlier findings if it has to hold them all in context. So the skill writes everything to .bugaudit/ as it goes: an append-only findings.jsonl (one bug = one JSON line, with a fixed schema), an inventory.md, a static-analysis.md, and a notes.md progress log. Phase 6 (triage/dedup) then becomes a mechanical "read all lines, group by root cause" operation instead of relying on recall.

Three independent levers for "thinking mode max". Different tools expose reasoning depth differently, and none of them are guaranteed to be present in every version, so the skill stacks three layers (detailed in §6) rather than betting on one.

Read-only by default. A bug-finding skill that also starts editing files is a different (riskier) product. This skill explicitly never modifies source files; "fix the bugs" is a deliberate follow-up task, not part of this skill.

3. Compatibility matrix & file layout

Tool Skill directory the tool scans Explicit invocation Notes
Claude Code .claude/skills/<name>/SKILL.md (project) or ~/.claude/skills/<name>/SKILL.md (personal) /find-bugs Supports extra frontmatter: effort, allowed-tools, context: fork. Skills are auto-exposed as slash commands using the skill's name.
OpenCode .opencode/skill/<name>/SKILL.md and/or .opencode/skills/<name>/SKILL.md (naming has varied across releases) /find-bugs (skills are auto-registered as commands) Also has a separate .opencode/command/ mechanism for non-skill slash commands — not needed here since the skill itself becomes a command.
Codex CLI .agents/skills/<name>/SKILL.md at the repo root (the open Agent Skills standard location); can be disabled per-skill via ~/.codex/config.toml $find-bugs mention, or /skills then pick it Also reads AGENTS.md (and AGENTS.override.md) for always-loaded project context — used here as a universal fallback pointer.
Other / generic CLI agents (Cursor, Gemini CLI, Copilot, etc.) Varies; many follow the same SKILL.md standard, or fall back to AGENTS.md Varies The .agents/skills/ copy plus the AGENTS.md pointer give these tools the best chance of picking it up even without dedicated support.

BUILD_GUIDE.md therefore writes one canonical copy to .agents/skills/find-bugs/ and mirrors it (full directory copy, not a symlink, for cross-platform/git-friendliness) into .claude/skills/, .opencode/skill/, and .opencode/skills/, then adds a short pointer block to AGENTS.md.

.agents/skills/find-bugs/        <- canonical
├── SKILL.md
├── references/bug-taxonomy.md
├── assets/report_template.md
└── scripts/run_static_checks.sh

.claude/skills/find-bugs/        <- mirror (identical contents)
.opencode/skill/find-bugs/        <- mirror (identical contents)
.opencode/skills/find-bugs/        <- mirror (identical contents)

AGENTS.md                         <- pointer block appended

Mirror-sync invariant. The three mirror directories (.claude/, .opencode/skill/, .opencode/skills/) are byte-identical to .agents/skills/find-bugs/ and must stay that way. Any edit to the canonical copy should be followed by re-running BUILD_GUIDE.md Step 6 (or its cp -r line in isolation) to re-mirror. Future maintainers: if you add a scripts/sync-mirrors.sh helper, document it here.

4. Activation design (the description field)

description is the only field every tool uses for automatic matching, so it carries the most weight in this spec. Design rules followed:

  • Front-load the exact target phrase. Codex's own guidance is to front-load key trigger words because descriptions can be truncated in the initial skills list. find-bugs's description opens with the literal phrase "find bugs in the codebase".
  • Enumerate near-synonyms as keywords, not as a grammatical sentence — "audit/scan/review the code for bugs or issues, code health check, debug the repository" — so semantic matching catches paraphrases.
  • Name all six categories ("logical errors, typos, formula/calculation mistakes, inconsistencies, runtime errors, compile-time/type/syntax errors") so a request like "check this repo for type errors" also matches, even if the user never says "bug".
  • State the output artifact ("write a categorized BUG_REPORT.md") so the description itself documents the contract.

Explicit invocation (/find-bugs, $find-bugs, /skills) is unaffected by description quality and always works as a fallback if a phrasing doesn't trigger automatically — BUILD_GUIDE.md Step 9 tells the user to fall back to this if needed.

5. Bug taxonomy (summary)

Full checklists and examples live in references/bug-taxonomy.md. Summary:

# Category One-line definition
1 Logical Control flow / boolean logic doesn't match the intent implied by naming, comments, or surrounding code.
2 Typos A misspelled literal, key, identifier, or pattern that should byte-for-byte match another one but doesn't, causing a silent mismatch.
3 Formula / Calculation Arithmetic, statistical, or unit-conversion logic that runs fine but computes the wrong number.
4 Inconsistency Two or more places that should agree (behavior, format, naming, validation, docs) but have drifted apart.
5 Runtime Code that parses/compiles fine but can throw, crash, hang, or behave unsafely on realistic inputs.
6 Compile-time / Syntax / Type The code would fail to compile/parse or be rejected by a type checker — highest-confidence, overlaps with Phase 2's automated tools.

These six map directly to the six section headers in BUG_REPORT.md and the six category values in the findings.jsonl schema (logical, typo, formula, inconsistency, runtime, compile).

6. "Thinking mode max" — the three-layer strategy

Layer Mechanism Where it lives Applies to
1 effort: max in SKILL.md frontmatter .agents/skills/find-bugs/SKILL.md (and mirrors) Claude Code's adaptive-effort system (mid-2026): skill/subagent frontmatter sets the effort level for the duration of that skill, overriding the session default (though not an explicit CLAUDE_CODE_EFFORT_LEVEL env var or --effort flag, which take precedence if set).
2 The literal word ultrathink inside the SKILL.md body's "Reasoning depth" section Same file, §0 of SKILL.md Claude Code's trigger-word preprocessing, which scans prompt text for think / think hard / think harder / ultrathink and sets a thinking-token budget accordingly — a second, independent lever in case effort isn't read for some reason.
3 Explicit, model-agnostic instructions: "set your reasoning-effort/thinking-budget to the highest available level... work through every phase with explicit step-by-step reasoning... externalize work to scratch files" Same section Any model/tool, including Codex CLI (where the equivalent lever is a model_reasoning_effort config setting, mentioned by name as a hint to the user/operator) and OpenCode, and DeepSeek-class models that respond to explicit reasoning instructions even without a formal "effort" API.

Why three layers instead of relying on the user typing "ultrathink": the skill is supposed to auto-trigger on a plain request like "find bugs in the codebase", which won't contain any thinking keyword. The frontmatter and in-body instructions make the skill self-sufficient regardless of how it was invoked.

Caveat: effort-level systems and exact config keys evolve quickly. If effort: max is rejected or ignored by a given tool/version, it's an unknown YAML key and is simply ignored (no parse error) — Layers 2 and 3 still apply. If you're building/maintaining this skill later and find a newer or different mechanism, update §0 of SKILL.md accordingly; the rest of the skill doesn't depend on which mechanism is current.

7. Pipeline overview

SKILL.md §3 defines eight phases; this is the one-paragraph version of each — see SKILL.md for the authoritative, detailed instructions.

  • Phase 0 (Setup): create .bugaudit/ scratch directory and its four files; gitignore it.
  • Phase 1 (Inventory & triage): enumerate files, detect languages/tools, classify every file into Tier 1 (always deep-reviewed: entry points, core logic, anything calculation-shaped), Tier 2 (reviewed as budget allows), or Tier 3 (tests/fixtures/migrations/generated — skimmed only). Flags oversized repos (>~150 Tier 1+2 files) up front instead of silently giving partial coverage.
  • Phase 2 (Static analysis): run scripts/run_static_checks.sh (or equivalent ad-hoc commands) for whatever toolchains are detected — this is the cheapest, highest-confidence source of compile/runtime findings.
  • Phase 3 (Per-file deep review): the core LLM-reasoning pass. Every Tier 1/2 file is checked against all six categories from bug-taxonomy.md; each issue becomes one JSON line in findings.jsonl.
  • Phase 4 (Cross-file consistency): a second pass, this time across files — signature/call-site mismatches, drifted duplicates, API-contract mismatches, stale docs.
  • Phase 5 (Formula verification): a targeted third pass over anything with business/scientific arithmetic — derive the intended formula, compare term-by-term, check units/percentages/rounding/aggregation.
  • Phase 6 (Triage/dedup/severity): mechanical pass over findings.jsonl — merge same-root-cause findings, confirm severities, drop false positives, split low-confidence items into an appendix.
  • Phase 7 (Write the report): fill assets/report_template.md with the triaged findings.
  • Phase 8 (Chat summary): a short human-readable summary — never a re-paste of the full report.

8. Severity rubric

Severity Meaning
Critical Crash, data loss/corruption, security issue, or completely wrong output on a common/primary path.
High Incorrect results/failures on common inputs — materially wrong, not necessarily a crash.
Medium Wrong only on edge cases, or a real inconsistency likely to cause a future bug.
Low Cosmetic — typo in a comment/log/UI string, minor style inconsistency, very-low-probability edge case.

confidence (high/medium/low) is tracked separately from severity — a finding can be severe but uncertain (goes to the "Possible Issues" appendix) or minor but certain (goes in the main tables as Low).

9. BUG_REPORT.md schema

Defined verbatim in assets/report_template.md; structurally:

  1. Header metadata (repo, date, scope, files reviewed/skipped).
  2. Summary table — category × severity counts.
  3. "Top Priority Issues" — 3-10 line summary of the worst findings.
  4. Sections 1-6, one per bug category, each finding as its own ### N.M [Severity] Title - file:lines subsection with What / Why it's a bug / Suggested fix / Confidence.
  5. "Possible Issues / Needs Verification" — low-confidence findings.
  6. "Static Analysis Output" — summarized tool results.
  7. "Methodology & Limitations" — exclusions, tools available/unavailable, unreviewed files, assumptions.

Every category section must be present even if empty (write "No issues found in this category.") so the report's shape is predictable for any downstream tooling or for a human skimming it.

10. Acceptance / validation checklist

After running BUILD_GUIDE.md, confirm:

  • .agents/skills/find-bugs/SKILL.md exists and begins with valid YAML frontmatter containing name: find-bugs, a non-empty description, and effort: max.
  • references/bug-taxonomy.md and assets/report_template.md exist and are non-empty.
  • scripts/run_static_checks.sh exists, is executable (chmod +x applied), and running it against an empty directory prints the header/footer lines without erroring.
  • .claude/skills/find-bugs/, .opencode/skill/find-bugs/, and .opencode/skills/find-bugs/ each contain the same four files — spot-check with diff -r .agents/skills/find-bugs .claude/skills/find-bugs (should print nothing).
  • AGENTS.md ends with the "Bug audit skill" block.
  • Smoke test: run the skill (explicitly, if auto-trigger doesn't fire) on a real or small test repo. Confirm BUG_REPORT.md is created at the repo root with all six category headers present, a filled-in summary table, and a "Methodology & Limitations" section. Confirm .bugaudit/ was created and contains findings.jsonl.
  • No side effects: run git status before and after — the only new paths should be BUG_REPORT.md, .bugaudit/ (if not gitignored yet), and whatever Step 6/7 added during the build. The skill itself must not touch any existing source file.

Optional deeper smoke test (seeded bugs)

If you want concrete proof each category is detected, create a throwaway file with one deliberate bug per category and re-run the skill against it. Examples (pseudocode, adapt to any language):

  • Logical: if (count > 10) where the comment says "trigger when 10 or more" (should be >=).
  • Typo: compare status == "complete" against a constant defined as "completed".
  • Formula: total = price - price * 20 where 20 should be 0.20 (20% discount).
  • Inconsistency: two near-identical helper functions where only one handles a None/null name.
  • Runtime: items[0].name on a list that can be empty.
  • Compile/type: call a function with one fewer argument than its definition requires.

A correct run produces one finding per category referencing the right line.

11. Customization knobs

  • Output filename/location: change the default in SKILL.md §1 (and assets/report_template.md's title if you want it reflected there too).
  • Large-repo threshold: the "~150 files" figure in Phase 1 step 5 is a judgment call — raise it for a faster but shallower default, lower it to force more explicit "Methodology & Limitations" disclosure on medium-sized repos.
  • Tier 1 patterns: the entry-point/core-directory/calculation-keyword globs in Phase 1 step 3 are intentionally generic; add project-specific patterns (e.g. a monorepo's package names) directly in SKILL.md if this skill will live in one specific repo rather than being copied around.
  • Static-analysis coverage: scripts/run_static_checks.sh covers Node/TS, Python, Go, Rust, Java/Kotlin, .NET, PHP, and Ruby. Add another if [ -f <manifest> ]; then ... fi block following the existing pattern for any other ecosystem (each block must remain self-guarded with command -v checks and a timeout, and must not auto-fix/format).
  • Findings schema: if you add fields to the findings.jsonl schema (e.g. a tags array), update both SKILL.md §3.3 (where the schema is documented) and assets/report_template.md if the new field should surface in the report.

12. Known limitations

  • Very large monorepos (thousands of files) will hit the Tier-1/Tier-2 budget quickly; the skill is designed to disclose this rather than solve it, but for true monorepos consider scoping invocations to one package at a time (the skill already supports a user-specified path).
  • Formula verification (Phase 5) depends on domain inference — for highly specialized math (actuarial, cryptographic, ML-numerical) the model may not know the "correct" formula to compare against; such findings should naturally land at confidence: low rather than being asserted confidently.
  • Static-analysis script coverage is best-effort and assumes common CLI tool names/flags; some projects use wrapper scripts (make lint, npm run check) instead — Phase 2 step 2 already tells the agent to fall back to ad-hoc commands if the bundled script doesn't cover the project's toolchain.
  • No fix mode. By design. A natural follow-up skill (fix-bugs or apply-bug-fixes) could read BUG_REPORT.md/findings.jsonl and apply fixes one at a time with user confirmation — out of scope here.

13. Relationship between the three deliverables

  • SPEC.md (this file) — the why. Read once when building or modifying the skill.
  • BUILD_GUIDE.md — the what, as literal, runnable shell commands. Source of truth for file contents and paths. Safe to re-run (idempotent: mkdir -p, cat > overwrite, rm -rf before cp -r).
  • PROMPT_FOR_AGENT.md — the short instruction to hand to a coding agent (e.g. DeepSeek V4 Flash) so it executes BUILD_GUIDE.md mechanically, without reinterpreting or "improving" it.

None of these three files are part of the skill itself — once built, the skill is just .agents/skills/find-bugs/ (+ mirrors + the AGENTS.md pointer). SPEC.md and BUILD_GUIDE.md can be deleted from the target repo after a successful build, or kept under e.g. docs/dev/ for future maintenance.