|
| 1 | +--- |
| 2 | +status: draft |
| 3 | +date: 2026-06-14 |
| 4 | +slug: deep-audit |
| 5 | +supersedes: null |
| 6 | +superseded_by: null |
| 7 | +pr: null |
| 8 | +outcome: null |
| 9 | +--- |
| 10 | + |
| 11 | +# Design: Full-codebase deep audit (perf · security · refactoring · bugs) |
| 12 | + |
| 13 | +## Summary |
| 14 | + |
| 15 | +Run a fresh full-codebase deep audit of `httpware` (code + tests + |
| 16 | +`architecture/` docs) using a multi-agent Workflow, producing one findings |
| 17 | +report at `planning/audits/2026-06-14-deep-audit.md` in the established |
| 18 | +taxonomy (Blocker / High / Medium / Low / Nit + a Negative-results section). |
| 19 | +The audit deliberately covers the dimensions the |
| 20 | +[2026-06-07 deep audit](../../../audits/2026-06-07-deep-audit.md) explicitly |
| 21 | +left uncovered — **performance, security, supply-chain** — plus correctness, |
| 22 | +concurrency, refactoring, and test quality. **Report only: no code changes, |
| 23 | +no fix PRs.** Confirmed findings spawn follow-up change bundles later, per the |
| 24 | +normal audit→fix flow. |
| 25 | + |
| 26 | +## Motivation |
| 27 | + |
| 28 | +- The 2026-06-07 deep audit states in its own summary: *"No dedicated chunk |
| 29 | + covered performance, security, or supply-chain dimensions"* and its `tests` |
| 30 | + dimension stalled (~1.5M Sonnet tokens, zero findings). Those are real |
| 31 | + coverage gaps in the only full audit on record. |
| 32 | +- Since 0.8.0 the codebase has grown materially (sync `Client`, circuit |
| 33 | + breaker, async timeout, multi-decoder routing, per-instance decoder cache). |
| 34 | + Delta audits covered each in isolation; nothing has swept the whole surface |
| 35 | + with the gap dimensions in mind. |
| 36 | +- The existing harness (`planning/audits/scripts/workflow.mjs`) still encodes |
| 37 | + pre-restructure reality: it points finders at `docs/*.md` and |
| 38 | + `planning/engineering.md` (now `architecture/*.md` and `planning/changes/`), |
| 39 | + pins stale model IDs (`claude-opus-4-7`), and has no performance, security, |
| 40 | + or refactoring finders. A fresh run needs an updated orchestrator. |
| 41 | + |
| 42 | +## Non-goals |
| 43 | + |
| 44 | +- No code changes, fixes, or PRs — this pass produces a report only. |
| 45 | +- No planning-doc staleness sweep beyond `architecture/` (the recent |
| 46 | + docs/UX work already churned `planning/` and the docs site). |
| 47 | +- No re-audit of the docs site content (`docs/`) — it had its own |
| 48 | + 2026-06-13 docs audit. |
| 49 | +- Not a delta audit — scope is the whole `src/httpware/**` + `tests/**` |
| 50 | + surface, not a single version's diff. |
| 51 | + |
| 52 | +## Design |
| 53 | + |
| 54 | +### 1. New orchestrator: `workflow-deep.mjs` |
| 55 | + |
| 56 | +Fork `planning/audits/scripts/workflow.mjs` into a sibling |
| 57 | +`workflow-deep.mjs` rather than mutating the existing (delta-oriented) script. |
| 58 | +Same four-phase pipeline — **Discover → Find → Verify → Synthesize** — same |
| 59 | +schemas (`FINDING_SCHEMA`, `VERDICT_SCHEMA`, `DISCOVER_SCHEMA`). Differences: |
| 60 | + |
| 61 | +- **Model IDs refreshed:** `claude-opus-4-8` for discover + synthesis, |
| 62 | + `claude-sonnet-4-6` for finders + verifiers. |
| 63 | +- **Paths corrected** to current repo reality everywhere they appear in |
| 64 | + prompts: `architecture/*.md` (not `docs/*.md`), `planning/changes/` (not |
| 65 | + `planning/engineering.md` / `planning/specs/`), `CLAUDE.md` invariants. |
| 66 | +- **Single combined run** (not per-chunk): all finders fan out in one |
| 67 | + `parallel()`, one verify pass, one synthesis writing the whole report. |
| 68 | + At ~11.8K LOC the synthesis context stays manageable, and a single run is |
| 69 | + simpler to reason about than the old chunk-and-commit-per-chunk flow. |
| 70 | +- **Discover refresh:** rebuild the module map to a dated file |
| 71 | + `planning/audits/scripts/_discover-2026-06-14.json` so the run is |
| 72 | + reproducible and doesn't clobber the existing `_discover.json`. |
| 73 | + |
| 74 | +### 2. Finder dimensions (one agent each) |
| 75 | + |
| 76 | +The four selected areas expand to ten focused finders so each stays narrow |
| 77 | +and adversarial (broad finders dilute signal): |
| 78 | + |
| 79 | +| Area | Finders | Status | |
| 80 | +|------|---------|--------| |
| 81 | +| Correctness & concurrency | `correctness`, `concurrency` (sync/async parity), `error_contract` | reuse, path-refreshed | |
| 82 | +| Performance | `performance` | **new** | |
| 83 | +| Security & supply-chain | `security` | **new** | |
| 84 | +| Refactoring & test quality | `refactoring`, `tests`, `public_api`, `optional_extras` | `refactoring` new; rest reused | |
| 85 | +| Architecture docs | `architecture_docs` (drift of `architecture/*.md` vs code) | reuse, repointed from `docs`/`planning_docs` | |
| 86 | + |
| 87 | +Each finder: reads the discover map first, targets 6–12 high-signal findings, |
| 88 | +returns the `FINDING_SCHEMA` (dimension, title, file, line_hint, claim, |
| 89 | +evidence_quote, suspected_severity, reproducer_hint), and **defaults to |
| 90 | +silence when uncertain** (quality > quantity). Per-dimension cap stays at 15. |
| 91 | + |
| 92 | +New finder prompt sketches: |
| 93 | + |
| 94 | +- **`performance`** — hot-path allocations and redundant work in the |
| 95 | + middleware chain composition and per-request `send`; lock-hold scope and |
| 96 | + contention in `RetryBudget` / `Bulkhead` / `CircuitBreaker`; decoder / |
| 97 | + `TypeAdapter` caching effectiveness (the per-instance cache landed in 0.9.0); |
| 98 | + avoidable async overhead (gather vs sequential, event-loop-blocking calls); |
| 99 | + unnecessary `Response` body reads / copies. Out of scope: correctness, |
| 100 | + concurrency *hazards* (those are other finders) — this is cost, not safety. |
| 101 | +- **`security`** — untrusted-response handling (status/header/body trust |
| 102 | + boundaries); decoder deserialization safety (pydantic/msgspec on attacker- |
| 103 | + controlled bytes, recursion/size limits); header, redirect-following, and |
| 104 | + URL/SSRF surfaces inherited from `httpx2`; exception messages or logs that |
| 105 | + could leak secrets (auth headers, URLs with credentials); dependency pinning |
| 106 | + and the optional-extras supply-chain surface (version floors in |
| 107 | + `pyproject.toml`). Out of scope: pure logic bugs. |
| 108 | +- **`refactoring`** — duplication between sync and async surfaces that could |
| 109 | + share a helper without crossing a protocol seam; inconsistent naming / |
| 110 | + signatures / error-construction patterns; dead code; over-complex control |
| 111 | + flow; module-boundary smells. Findings are *suggestions* with a stated |
| 112 | + payoff, never style nits dressed up as bugs. Default severity low/nit unless |
| 113 | + the duplication has caused a real divergence. |
| 114 | + |
| 115 | +Reused finders (`correctness`, `concurrency`, `error_contract`, `tests`, |
| 116 | +`public_api`, `optional_extras`, `architecture_docs`) carry forward their |
| 117 | +2026-06-07 prompts verbatim except for the path/model corrections in §1. |
| 118 | + |
| 119 | +### 3. Verify — 3-lens panel per finding |
| 120 | + |
| 121 | +Unchanged from the existing harness. Each surviving candidate gets three |
| 122 | +independent Sonnet verifiers: |
| 123 | + |
| 124 | +- **`code_reality`** — re-read the cited code ±30 lines; did the finder |
| 125 | + misread? Default `confirmed: false` if the code doesn't support the claim or |
| 126 | + can't be located. |
| 127 | +- **`reproducer`** — sketch a 3–5 line test that demonstrates the bug (or, for |
| 128 | + a doc finding, a reasonable misleading read). No repro ⇒ `confirmed: false`. |
| 129 | +- **`spec_grounded`** — does it violate a stated `CLAUDE.md` invariant (error |
| 130 | + contract, optional-extras isolation, no `httpx2._`, no global logging, |
| 131 | + naming)? Raise severity if it hits a listed invariant; lower if the spec is |
| 132 | + silent and it's a hardening suggestion. |
| 133 | + |
| 134 | +A finding **survives on ≥2/3 confirms**. Severity is lowered if ≥1 verifier |
| 135 | +says lower, raised if ≥2 say raise (existing `lowerOne`/`raiseOne` logic). |
| 136 | + |
| 137 | +### 4. Synthesize — one report, with Negative results |
| 138 | + |
| 139 | +A single Opus synthesis agent: |
| 140 | + |
| 141 | +1. Triages surviving findings into Blocker / High / Medium / Low / Nit using |
| 142 | + the spec's severity definitions; rolls up >4 nits per dimension into one |
| 143 | + entry. |
| 144 | +2. Dedups across dimensions (file + line ±5 + similar claim). |
| 145 | +3. Writes `planning/audits/2026-06-14-deep-audit.md`: top Summary |
| 146 | + (counts + headline), per-bucket findings (each: `file:line` in code format, |
| 147 | + ≤3-sentence claim, fenced evidence quote, verifier consensus e.g. "3/3: |
| 148 | + code_reality, reproducer, spec_grounded", suggested direction), and a |
| 149 | + **Negative results** section. |
| 150 | +4. **Negative results** are built from candidates the panel *refuted* |
| 151 | + (`surviving === false`) plus invariants the finders explicitly checked and |
| 152 | + found held — the "verified-correct surface" that is a signature of the |
| 153 | + prior reports. The new orchestrator passes the refuted candidates to |
| 154 | + synthesis instead of silently dropping them (the one structural change from |
| 155 | + the old harness, which discarded non-survivors). |
| 156 | +5. Commits the report with an `audit(deep): …` message. **No source edits.** |
| 157 | + |
| 158 | +## Operations |
| 159 | + |
| 160 | +None. Entirely in-repo; no infra, DNS, or external accounts. |
| 161 | + |
| 162 | +## Out of scope |
| 163 | + |
| 164 | +Listed under Non-goals. The one explicit follow-up: confirmed findings feed |
| 165 | +the normal audit→fix flow as new `planning/changes/active/` bundles, triaged |
| 166 | +by severity, in a *separate* session. |
| 167 | + |
| 168 | +## Testing |
| 169 | + |
| 170 | +This change ships an audit report + an orchestrator script, not library code, |
| 171 | +so "testing" means validating the *process*, not pytest: |
| 172 | + |
| 173 | +- The Workflow completes all four phases without an unhandled throw; the |
| 174 | + discover JSON is written; the report file exists and parses as the expected |
| 175 | + Markdown structure (all five severity buckets present even if empty, a |
| 176 | + Negative-results section present). |
| 177 | +- Spot-check: manually reproduce the single highest-severity finding (if any) |
| 178 | + to confirm the panel didn't pass a false positive — matching the prior |
| 179 | + audits' "headline findings reproduced directly" discipline. |
| 180 | +- Sanity: total confirmed count is plausible (not 0 across all ten finders, |
| 181 | + which would signal a broken discover map or mispointed paths — the failure |
| 182 | + mode that stalled the 2026-06-07 `tests` dimension). |
| 183 | + |
| 184 | +## Risk |
| 185 | + |
| 186 | +- **Stale paths silently yield zero findings** (most likely × high impact): |
| 187 | + if a finder prompt still points at a moved/renamed file it returns nothing |
| 188 | + and looks "clean." Mitigation: every path in every prompt is rewritten in §1 |
| 189 | + against the verified current tree; discover map is regenerated fresh and |
| 190 | + finders are required to read it before searching. |
| 191 | +- **False positives surviving the panel** (medium × medium): three Sonnet |
| 192 | + verifiers can collectively confirm a plausible-but-wrong finding. |
| 193 | + Mitigation: `code_reality` defaults to false on any doubt; headline findings |
| 194 | + are hand-reproduced before the report is trusted. |
| 195 | +- **Token cost overrun** (medium × low): ~10 finders + (~80 candidates × 3 |
| 196 | + verifiers) + synthesis ≈ 250 mostly-Sonnet agents, Opus only for discover + |
| 197 | + synthesis — in line with the prior ~1M-token deep audit, which the user has |
| 198 | + opted into. The 15/dimension cap and ≥2/3 gate bound the verify fan-out. |
| 199 | +- **Refactoring finder produces noise** (medium × low): subjective cleanup |
| 200 | + suggestions can crowd the report. Mitigation: prompt forces a stated payoff |
| 201 | + and low/nit default; the spec_grounded lens lowers anything the conventions |
| 202 | + are silent on. |
0 commit comments