Skip to content

V33 self-test smoke #4: 5.7% recall — 4 probe-wiring blockers #95

@cunninghambe

Description

@cunninghambe

Status

First smoke run that completed end-to-end (178/178 tests, no crashes). Pipeline infrastructure is now healthy. Remaining gap is structural probe wiring.

Numbers

  • Recall: 6/105 golden kinds detected (5.7%)
  • False positives: 0
  • Pipeline: clean (no crashes, all tests ran)

Hits (6)

  • `coop_coep_violation` (28)
  • `focus_lost_after_action` (28)
  • `missing_state_change` (12)
  • `seo_h1_missing_or_multiple` (2)
  • `seo_title_duplicate_across_routes` (1)
  • `xss_reflected` (1)

Blockers (ordered by impact)

1. Pen-testing and race modules probe 0 targets

Both modules enabled, both ran `probesAttempted: 0 / testsAttempted: 0`. Root cause: `surface_list_tools` returned 0 tools because the fixture's API server (`self-api` on 5791) is not registered as a SurfaceMCP surface. Only the `self-spa` surface is in `/tmp/surfacemcp-fixture.json`.

Fix: add the `self-api` surface (stack: `express`, port 5791) to the SurfaceMCP fixture config and register its routes.

Unblocks: `sql_injection`, `command_injection`, `path_traversal`, `jwt_weak_alg`, `open_redirect`, `xss_dom`, all race kinds, all IDOR kinds (~25 BugKinds total).

2. HAR network capture empty

`discovery.probe.telemetry.probesRun: 0`, HAR files empty. BrowserMCP network intercept not functioning against Vite dev server.

Unblocks: `network_5xx`, `console_error`, `react_error`, `n_plus_one_api_calls` (~15 BugKinds).

3. Bundle probe returns 0 bytes

`initialJsBytesGzipped: 0` — bundle analyzer not reaching the Vite dev build's asset manifest.

Unblocks: `oversized_bundle` and related perf kinds.

4. Axe instrumentation silent

28 pages crawled, `a11yStrict: true`, but axe runner emitted 0 results. Bootstrap is silently failing.

Unblocks: `form_input_unlabeled`, `image_missing_alt`, `keyboard_trap`, `touch_target_too_small`.

Why this is good news

Before today the smoke crashed before producing any clusters (lockstep / IPv6 / fixture config / stale .js — all fixed in PRs #89, #91, #92). We now have a measurable baseline (6/105) that future probe-wiring fixes can move incrementally. Each of the four blockers is a discrete, well-scoped follow-up rather than an unknown.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions