agentprobe

Your docs pass review because the reviewer already knows the answer.

agentprobe sends an AI agent through them cold — in a Docker sandbox, with no prior knowledge — and reports every place it got stuck, with file:line and a suggested fix.

Open-source · local-first · BYOK · MIT

What it found in the wild

We pointed agentprobe at three of the most-complained-about docs in open source — LangChain, boto3, and FFmpeg — using nothing but the generic built-in tasks (quickstart, auth-setup, error-handling). No hand-tuning, no per-SDK config, about 45 minutes of wall time. All three came back 0/3 on pass rate.

LangChain — three READMEs fighting each other. The root README.md presents itself as primary. libs/langchain/README.md calls itself "langchain-classic" and says it's legacy. libs/langchain_v1/README.md also calls itself "LangChain". All three tell you to pip install langchain, and no page explains the relationship. This is the v0 / v1 / classic confusion the community has been complaining about for over a year — surfaced cold, in one run, by an agent that had never read the complaints.

boto3 — docs/source/guide/quickstart.rst:112. The "where to obtain credentials" section bottoms out in external IAM Console links and stops: no aws configure, no ~/.aws/credentials sample, no export AWS_ACCESS_KEY_ID=…. The agent's notes truncate mid-sentence, because there was nothing left in the SDK to read.

FFmpeg — doc/protocols.texi:1. 2347 lines of accurate protocol reference, with authentication scattered across the AMQP (line 66), FTP (376), and HTTP (512) sections and no index or entry point. The agent read top-to-bottom trying to synthesize a unified guide and hit its output limit at "AMQP with Au". A human ctrl-Fs their protocol in 30 seconds; an agent without that meta-knowledge drowns.

The line where docs die

Three unrelated repos, three languages, one shared failure mode — the example ends one line too early:

Repo	Location	What's missing
LangChain	README.md:37	no print() after model.invoke()
boto3	README.rst:88	no expected output after the S3 list-buckets call
FFmpeg	doc/examples/show_metadata.c:24	no sample key=value output

Humans skim past this because they already know what model.invoke / list_buckets / metadata extraction should print. An agent has no such intuition: it writes the code, runs it, and has nothing to compare the result against. The line between "the code" and "did it work" is where docs reliably die — and it's the line agentprobe is built to find.

Honest accounting: across those three runs the judge emitted 15 issues and 7 survived our own triage. The discards were mostly duplicates of a pattern already reported against another repo, plus one premise mismatch — the auth-setup task assumes account-level authentication, which FFmpeg simply doesn't have. Expect to skim a shortlist, not to merge a patch series unread. See Limitations.

What it looks like

You point it at a repo and pick the tasks you want exercised:

agentprobe scan --repo https://github.com/stripe/stripe-node --tasks quickstart,auth-setup

The sandbox boots, Claude Code runs each task, the judge reads the resulting trace, and you get a markdown report:

## stripe-node — auth-setup (partial)

### Issue 1 — missing_step  (confidence 0.85, Critical)
**README.md:42** — also: README.md:50

The agent answered "Partial" on the validation question because no
section describes how to verify a freshly-set API key. The setup section
ends at the `customer.create` example with no "if this prints a cus_…
ID, auth works" line.

**Suggested fix**: insert after line 51:
> ### Verifying your authentication
> Run `await stripe.balance.retrieve()` …

Each issue carries a category (missing_step, ambiguous_choice, naming_inconsistency, stale_doc, unclear_error, implicit_assumption), a primary file:line, optional also_consider locations, and a one-paragraph suggested fix grounded in real lines of your file.

How it's different

Most "AI for docs" tools either grade your docs in the abstract or rewrite them with an LLM and call it a day. agentprobe does the opposite:

• Real agent, real sandbox — spawns Claude Code in a Docker container, points it at your SDK, and gives it a task (quickstart, auth setup, error handling, etc.). The trace records every Read / Bash / Grep / Edit it makes while wrestling with your docs
• Three-stage judge, not one prompt — Haiku classifies the trace and locates friction events, Sonnet pinpoints the root cause per friction (file:line, category), Sonnet writes the suggested fix. Each stage has a narrow job; structured-output tool_use keeps the schema clean
• Reproducible eval set — 15 frozen traces under eval/traces/ with hand-labeled ground truth issues under eval/cases/. agentprobe eval re-runs the judge on those traces and scores precision/recall against GT. Lets you iterate the judge prompts without paying for fresh sandbox runs
• BYOK LLM — set ANTHROPIC_API_KEY and pay Anthropic directly. No proxy, no relay
• MIT-licensed, no telemetry — fork it, audit it, run it on internal SDKs

Quick start

Requires Node 20+, Docker, and an Anthropic API key. Get a key at https://console.anthropic.com/settings/keys (account + billing required).

git clone https://github.com/agentprobe2026/agentprobe.git agentprobe
cd agentprobe
npm install
npm run build

Then either run via the local dist/cli.js or npm link to get the agentprobe binary on your PATH.

# 1. Set your key (or copy .env.example to .env and fill it in)
export ANTHROPIC_API_KEY=sk-ant-...

# 2. See what tasks are available
agentprobe list-tasks
# ↑ if this prints task names without an auth error, your key is valid

# 3. Run a scan against a public repo
agentprobe scan \
  --repo https://github.com/frida/frida-python \
  --tasks frida-hook-open \
  --out ./runs

# 4. Open the markdown report
$ cat runs/<timestamp>/report.md
# ↑ a report with Critical / Warning issues means the scan worked —
#   that's the tool's job. An empty issues list means the agent finished
#   the task without friction. Exit code 0 = scan completed; non-zero
#   = sandbox or judge failure (see Troubleshooting).

Each scan run writes its sandbox workdir, raw trace JSONL, and the rendered markdown report under runs/<timestamp>/. Pass --thorough to run the judge twice and union results (~2× cost, +5–10pp recall, smooths model variance).

Iterating the judge against the eval set

# Re-score the judge on every labeled eval case (no sandbox cost — uses frozen traces)
agentprobe eval

# Or just a few cases while tuning a prompt
agentprobe eval --only 005-stripe-auth,009-langchain-errors

You get per-case TP/FP/FN, per-category precision/recall, and per-case judge outputs under runs/eval-out/ for inspection.

Features

• Four built-in tasks — three generic ones run by default (quickstart, auth-setup, error-handling) plus one SDK-specific (frida-hook-open, a category-coverage smoke test) that you opt into via --tasks frida-hook-open. Each is a YAML under src/tasks/builtin/ and you can add your own
• Six docs-defect categories — naming_inconsistency, missing_step, ambiguous_choice, stale_doc, unclear_error, implicit_assumption. The judge picks one per issue and is rewarded for category-accurate matches by the scorer
• Confidence binning — every issue lands in Critical (≥ 0.8), Warning (0.6–0.8), or Drop (< 0.6). The report only shows Critical + Warning by default
• Pipeline-level rescue — when stage 1 (Haiku) under-emits frictions because the agent declared Validation per docs = Partial / No or wrote a non-empty ## Gaps section, the pipeline scans the agent's own output and injects synthetic frictions so stage 2 sees them
• Category hints from regex — paragraphs in the agent's output matching telltale ambiguous_choice / implicit_assumption wording get a [CATEGORY HINT: …] tag that pushes stage 2 off its missing_step default
• Location hints + fallback synthesis — file:line citations extracted from gap paragraphs are forwarded to stage 2 as [LOCATION HINT: …], and if stage 2 still returns null on a hinted friction the pipeline synthesizes the issue directly from the hint to preserve recall

Configuration

agentprobe is a CLI; configuration is per-invocation rather than a settings file. The pieces that change behavior:

Flag / env	Default	Notes
ANTHROPIC_API_KEY	required	Sent only to Anthropic. Loaded from .env if present
--repo <spec>	required for scan / collect	Git URL or local path. Local paths are mounted into the sandbox read-write
--tasks <ids>	all generic builtins	Comma-separated, e.g. quickstart,auth-setup. SDK-specific tasks (currently frida-hook-open) are skipped by default; pass them explicitly to opt in
--out <dir>	./runs	Where sandbox workdirs, traces, and reports land
--thorough	off	For scan: run the judge pipeline twice and union issues. Doubles API cost but recovers ~5–10pp recall lost to temp=0 model variance
--only <case-ids>	all eval cases	Comma-separated case ids for agentprobe eval
LINE_TOL (in src/eval/score.ts)	8	Match tolerance between judge line and GT line. Tightening makes the eval stricter

Models are pinned in src/judge/pipeline.ts (claude-haiku-4-5-20251001 for stage 1, claude-sonnet-4-5 for stage 2/3). Prompts live under src/judge/prompts/*.md and are hot-reloaded on every run — edit and re-run agentprobe eval without recompiling.

Architecture

┌────────────────────────────────────────────────────────────┐
│                       agentprobe CLI                       │
│         scan · collect · eval · list-tasks                 │
└──────────────────────────────┬─────────────────────────────┘
                               │
   ┌───────────────────────────┴───────────────────────────┐
   │                                                       │
   │  ┌──────────┐   ┌──────────┐   ┌──────────────────┐   │
   │  │  tasks   │   │  runner  │   │      judge       │   │
   │  └──────────┘   └────┬─────┘   └────────┬─────────┘   │
   │                      │                  │             │
   │              ┌───────┴──────┐   ┌───────┴──────────┐  │
   │              │   sandbox    │   │   3-stage        │  │
   │              │  (dockerode  │   │   pipeline       │  │
   │              │  + Claude    │   │   (Haiku→Sonnet  │  │
   │              │  Agent SDK)  │   │   →Sonnet)       │  │
   │              └──────┬───────┘   └────────┬─────────┘  │
   │                     │                    │            │
   │              ┌──────┴──────────┐   ┌─────┴────────┐   │
   │              │   trace.jsonl   │──→│   report.md  │   │
   │              └─────────────────┘   └──────────────┘   │
   └───────────────────────────────────────────────────────┘

• src/cli.ts — commander-based CLI entry point with scan, collect, eval, list-tasks subcommands
• src/runner/sandbox.ts — boots a Docker container, mounts the target repo, runs Claude Code via the Claude Agent SDK, streams events into trace.jsonl
• src/judge/pipeline.ts — three-stage judge orchestrator. Stage 1 classifies + locates friction with Haiku; stage 2 produces one root-cause issue per friction with Sonnet; stage 3 writes the suggested fix
• src/judge/prompts/*.md — system prompts for each stage, loaded fresh on every run for fast iteration
• src/eval/score.ts — greedy bipartite matcher between judge output and ground truth issues; computes per-category precision/recall and confidence-bin agreement
• src/report/markdown.ts — renders the per-task issue list into the markdown the user reads

How a docs issue gets emitted

1. Stage 1 (Haiku) classifies the trace as success / partial / failure and lists up to 8 friction event indices with one-line summaries. Cheap and fast.
2. Pipeline rescue — if the agent's output file declared ## Validation per docs = Partial / No or has a non-empty ## Gaps section but stage 1 emitted zero frictions, the pipeline scans the agent's output and injects synthetic frictions so the rest of the pipeline isn't starved.
3. Pipeline category hints — paragraphs whose wording smells like ambiguous_choice (e.g. "two different", "vs", "third option") or implicit_assumption (e.g. "format not documented", "implicit") get a [CATEGORY HINT: …] prepended; file:line citations get a [LOCATION HINT: …]. These steer stage 2 off its missing_step default.
4. Stage 2 (Sonnet) produces one issue per friction — picks category, primary_location with a concrete file:line (Rule 1 in src/judge/prompts/root-cause.md is strict about no line: 1 fallbacks except when an entire section is structurally absent), up to three also_consider locations, and confidence. Returns null if no verifiable defect.
5. Hint fallback synthesis — when stage 2 returns null on a friction that already carries both CATEGORY and LOCATION HINTs, the pipeline builds the issue directly from the hint so recall isn't bottlenecked by Sonnet's null-return tendency.
6. Stage 3 (Sonnet) writes the suggested fix with the file's actual line-numbered content as context. Quotes real lines, not invented ones.

Eval set

The eval set lives entirely under eval/:

• eval/traces/<id>.jsonl — one frozen trace per case, captured by agentprobe collect against a pinned commit of the target SDK
• eval/cases/<id>.json — ground truth: classification + a list of expected issues, each with { category, file, line, agent_did, why_agent_wrong, why_human_wouldnt, suggested_fix, confidence }

agentprobe eval re-runs the judge against the frozen traces and uses a greedy bipartite matcher (src/eval/score.ts) that matches a predicted issue to a GT issue when:

• file path matches (basename-forgiving), AND
• predicted line is within ±8 of GT line (LINE_TOL), AND
• categories are identical

Matching is greedy and prefers primary_location matches over also_consider matches. The scorer reports precision, primary_recall, any_recall, classification accuracy, and confidence-bin agreement.

To add a new case:

agentprobe collect \
  --repo https://github.com/<owner>/<repo> \
  --task quickstart \
  --label 016-<repo>-quickstart

This runs one sandbox pass, freezes the trace under eval/traces/, and writes an empty eval/cases/016-…json template you fill in by hand.

Limitations

• Anthropic-only judge — stage 1 hits Haiku, stages 2 / 3 hit Sonnet. Swapping providers means rewriting src/judge/pipeline.ts's callWithStructuredOutput to a provider-agnostic shim
• Tool_use determinism is approximate — Anthropic's temperature=0 is not bit-reproducible across runs. Per-case TP/FP/FN can drift by ±1 between runs of the same trace. Aggregate metrics are stable; case-level scores are not
• Docker required — there's currently no in-process fallback for environments where Docker isn't available
• Category coverage is uneven — missing_step and naming_inconsistency perform well; ambiguous_choice and implicit_assumption recall depends heavily on whether the agent explicitly worded the gap (regex + hint heuristics catch the common phrasings but not pure semantic-comparison cases)

Privacy

• The Docker sandbox sees only the target repo. Your local files don't enter the container unless you mount them with --repo <local-path>.
• agentprobe sends data to Anthropic only — the agent's reasoning during the sandbox phase, the trace excerpt + agent output files during the judge phase.
• No telemetry, no analytics. Logs are local to runs/ and eval/.

Development

Requirements: Node 20+, Docker (or Docker Desktop on Windows/macOS), Git.

git clone https://github.com/agentprobe2026/agentprobe.git agentprobe
cd agentprobe
npm install
npm run dev -- scan --repo <some-public-sdk>   # tsx watch-mode equivalent of the built CLI

Useful scripts

Script	Purpose
npm run dev -- <args>	Run the CLI through tsx without rebuilding (good for iterating on prompts / scoring code)
npm run build	tsc + copies prompt .md files and built-in task yaml into dist/
npm run typecheck	Type-check without emitting
npm test	Vitest suite (parser tests, scorer tests)

Repo layout

agentprobe/
├── src/
│   ├── cli.ts                CLI entry (commander)
│   ├── tasks/
│   │   ├── builtin/*.yaml    task definitions: prompt + success_checks
│   │   └── loader.ts
│   ├── runner/
│   │   ├── sandbox.ts        Docker container + Claude Agent SDK driver
│   │   └── trace.ts          JSONL event writer
│   ├── judge/
│   │   ├── pipeline.ts       3-stage orchestrator
│   │   ├── prompts/*.md      stage 1 / 2 / 3 system prompts
│   │   └── index.ts          schemas + types
│   ├── eval/
│   │   ├── score.ts          greedy bipartite matcher + reporters
│   │   ├── schema.ts         ground-truth case schema (zod)
│   │   └── collect.ts        `agentprobe collect` implementation
│   ├── report/markdown.ts    issue list → markdown
│   └── utils/logger.ts
├── eval/
│   ├── cases/*.json          ground-truth issues per (repo, task) pair
│   └── traces/*.jsonl        frozen agent traces — `agentprobe eval` consumes these
└── scripts/copy-assets.mjs   copies prompts + builtin yaml into dist on build

Troubleshooting

Missing or invalid API key

Symptom: the CLI exits with ANTHROPIC_API_KEY is not set (looked in .env and environment).

• Verify the key is exported in the same shell you're running agentprobe from: echo $ANTHROPIC_API_KEY. The value must start with sk-ant-.
• If you're using a .env file, make sure it sits next to where you invoke agentprobe (the CLI uses dotenv/config).
• Generate or rotate keys at https://console.anthropic.com/settings/keys.

Re-run any agentprobe command after fixing — agentprobe list-tasks is the cheapest smoke test.

Docker not running

Symptom: scan / collect errors with "failed to connect to the docker API …" or "no such image".

• Start Docker Desktop (macOS / Windows) or sudo systemctl start docker (Linux).
• First run also has to build the sandbox image (agentprobe-sandbox:latest). If the image is missing, build it once via docker build -t agentprobe-sandbox:latest docker/ from the repo root.

Cannot copy repo to a subdirectory of itself

Symptom: ERR_FS_CP_EINVAL: cannot copy <path> to a subdirectory of self.

You're scanning a directory and writing --out inside that same directory. Move --out outside the target repo, e.g. --out ../scan-runs.

Sandbox timeout or container OOM

Symptom: task ends early or container exits with non-zero before producing output files.

Long-running tasks may hit Docker's default memory / CPU limits. Raise them in Docker Desktop → Settings → Resources (give Docker ≥ 6 GB RAM for the typical task). If a task consistently times out, raise its timeout_seconds field in the YAML under src/tasks/builtin/.

Contributing

PRs welcome. The interesting surfaces:

• src/judge/prompts/*.md — iterate prompts, run agentprobe eval --only <few cases> to validate
• src/judge/pipeline.ts — hint heuristics, synthesis fallback, signal extraction
• eval/cases/*.json — add more labeled cases, especially for under-covered categories
• src/tasks/builtin/*.yaml — new task types

A change is "validated" when its eval-set numbers don't regress on the dev cases and ideally improve on the target case.

License

MIT — see LICENSE.