research: LLM-assisted dynamic probing module (--attacks llm)

[ksek87]

Problem

Static pattern matching and scripted attack chains (fuzzd's current approach) catch ~60% of attacks on the MCPTox benchmark. The remaining 40% require dynamic, adaptive probing that responds to the specific server's behavior — the approach MCPGuard (arXiv:2510.23673) demonstrates with a hacker/auditor/supervisor agent loop.

fuzzd's deterministic, reproducible design is a strength for CI but a ceiling for recall. An opt-in LLM-assisted module (--attacks llm --api-key <key>) could close the gap without compromising the default path.

Approach (MCPGuard-inspired)

Three-agent loop, each backed by a Claude API call:

1. Hacker agent — given the server's tool list, generates adversarial tool calls targeting the most likely vulnerability: credential theft, exfiltration, privilege escalation, confused deputy
2. Auditor agent — receives the tool responses and classifies findings: is the response evidence of a vulnerability, a false positive, or inconclusive?
3. Supervisor agent — decides whether to terminate (finding confirmed / budget exhausted) or continue probing (which tool to target next)

The loop is bounded by a configurable --llm-budget (number of API calls or cost ceiling).

Key design constraints

• Determinism: findings must be reproducible. Solution: log the full hacker/auditor exchange; the SARIF output includes the prompt/response that produced the finding
• Cost: default budget is conservative (e.g., 10 API calls); user opts in explicitly
• No external dependency in default build: gate behind --attacks llm; the LLM client is only compiled when --features llm is set
• API: use Claude claude-sonnet-4-6 (or claude-haiku-4-5 for cost-sensitive runs)

Distinction from #52 (Neural Semantic Layer)

Issue #52 is about using a neural embedding model to improve the static scanner's recall — no API key, fully local, deterministic. This issue is about using an LLM agent for dynamic probing of live servers — requires an API key, non-deterministic, high recall ceiling. They are independent.

Acceptance criteria (research phase)

• Proof-of-concept: hacker/auditor loop against the reference vulnerable server fixture
• Measure recall on MCPTox actual (how many does it catch that static scan misses?)
• Measure cost per server audit (API calls, tokens, wall time)
• Assess reproducibility: same server → same findings across runs?
• Go/no-go decision: is the recall gain worth the cost and complexity?

References

• MCPGuard: https://arxiv.org/abs/2510.23673
• MCP-Guard multi-stage defense: https://arxiv.org/html/2508.10991
• Claude API: Sonnet 4.6 for hacker/auditor, Haiku 4.5 for supervisor to control cost