Skip to content

Severity classifier returns 'info' on every cluster on real-world target #147

@cunninghambe

Description

@cunninghambe

Found in Aspectv3 smoke

All 126 clusters from the Aspectv3 real-world run came back as `severity: 'info'`. Even structurally-significant findings:

  • `missing_csp_header` (security): info
  • `main_thread_blocked` 363ms (perf): info
  • `seo_meta_description_missing` × 31 routes: info

The synthetic deliberate-bugs fixture surfaces clusters with mixed severities (smoke #12 had clusters across critical/major/minor/info). On Aspectv3 — none.

Hypothesis

The severity classifier (likely `packages/cli/src/classify/severity.ts` or in the registry's per-kind metadata) may:

  1. Default to `info` when it can't extract specific signals (e.g. confidence too low, signature missing fields)
  2. Expect signals that the synthetic fixture provides explicitly but a real app doesn't (like marker DOM IDs, fixture annotations)
  3. Be silently catching errors and falling back to info

What to investigate

  • Read the severity classifier
  • Sample one cluster from Aspectv3 (e.g. missing_csp_header) and trace why severity stayed at info instead of major (CSP missing on a clinical app is structurally major)
  • Compare to the same kind from the synthetic fixture — what input signals differ?

Fix

Once root cause is found:

  • If it's a fallback bug: fix the path
  • If it's a signal-extraction issue on real apps: enrich the classifier to derive severity from kind + occurrence count + suspectedFiles when fixture-specific signals are absent
  • Don't lower the bar for what counts as `major` — but `info` for a missing CSP is wrong

Priority

High — severity is the headline triage signal. If everything is info, BugHunter loses its prioritization value on real apps.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions