fix(docs-review): check related docs by name and directory, not just identifier grep

[csoceanu]

Summary

The docs-review skill finds stale docs by grepping for identifiers from the diff. This misses docs that describe the same feature area but don't mention any specific changed identifier — common when a PR adds new behavior rather than renaming existing behavior.

This PR adds a step to check whether documentation files covering the same feature exist based on file name and directory structure, and evaluate them as candidates even without grep matches.

Changes

• docs-review/SKILL.md step 4: after the identifier grep, also check for related doc files by name/directory match
• docs-review/SKILL.md step 5: handle candidates without grep hits by reading section headings in the quick-scan pass
• docs-currency.md: align the sub-agent instructions with the updated skill

Context

An impact analysis of 111 merged PRs found that the skill missed stale docs in 7 PRs (6% false negative rate). In most of these cases, the PR added new user-visible behavior and the related doc file wasn't found because the new identifiers didn't exist in existing docs. This change addresses that gap by matching code files to doc files by name and directory structure, not just identifier grep.

[github-actions]

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

See E2E testing guide for details.

[qodo-code-review]

PR Summary by Qodo

Improve docs-review: include related docs via name/directory matching

🐞 Bug fix 📝 Documentation 🕐 10-20 Minutes

AI Description

• Expand docs-review candidate selection beyond identifier grep to include related docs by
 path/name.
• Update quick-scan guidance to evaluate no-grep candidates via section headings.
• Align docs-currency sub-agent instructions with the updated docs-review workflow.

Diagram

graph TD
  A["PR diff"] --> B["Extract identifiers"] --> C["Grep docs"] --> D["Name/dir match"] --> E["Candidate docs"] --> F["Quick scan (context/headings)"] --> G["Verdicts"]

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Semantic matching (LLM/embeddings) between diff summary and docs

• ➕ Catches conceptually-related docs even when names/paths don’t align
• ➕ Reduces reliance on repository-specific directory conventions
• ➖ More complex and expensive to run; may require infra/supporting tooling
• ➖ Harder to make deterministic and explainable for reviewers

2. Explicit code-to-doc ownership mapping (metadata file/frontmatter)

• ➕ Highly deterministic; fewer false positives from heuristic matching
• ➕ Scales well once the mapping is maintained
• ➖ Ongoing maintenance cost and risk of drift
• ➖ Requires repo-wide adoption to be effective

3. Directory-based allowlist of “docs per module” rules

• ➕ Simple implementation; predictable behavior
• ➕ Keeps logic close to existing project structure
• ➖ Less flexible across repos with different layouts
• ➖ May miss cross-cutting docs that live outside the module’s doc folder

Recommendation: The PR’s approach (augmenting grep with name/directory-based candidate discovery, then using headings for no-grep quick scans) is the best low-complexity improvement for reducing false negatives. Semantic matching or explicit mappings could further improve recall, but they add significant operational or maintenance overhead that doesn’t seem justified for this incremental fix.

Files changed (2) +12 / -3

Documentation (2) +12 / -3

SKILL.mdAdd name/directory-based related-doc discovery and heading-based quick scan +9/-2

Add name/directory-based related-doc discovery and heading-based quick scan

• Extends the candidate-doc collection step to include documentation inferred by file name and directory structure, not only identifier grep matches. Updates the quick-scan pass to handle candidates without grep hits by reading section headings to determine relevance.

internal/scaffold/fullsend-repo/skills/docs-review/SKILL.md

docs-currency.mdAlign docs-currency instructions with related-doc discovery heuristic +3/-1

Align docs-currency instructions with related-doc discovery heuristic

• Updates the sub-agent guidance to also consider docs that cover the same feature area by name/directory match, even when no identifier references are found.

internal/scaffold/fullsend-repo/skills/pr-review/sub-agents/docs-currency.md

[github-actions]

Site preview

Preview: https://4c4fee58-site.fullsend-ai.workers.dev

Commit: 8ac17f5e2fb7c7aa22e3d5358534d77c670ef3d4

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📜 Skill insights (0)

Context used

✅ Compliance rules (platform): 61 rules

✅ Skills: writing-user-docs, writing-adrs

1. Re-adds PR-modified docs ✓ Resolved 🐞 Bug ≡ Correctness

Description

Step 4 excludes documentation files modified in the PR, but the newly added name/directory-match
step can add those same files back into the candidate list (because they’re no longer “already in
the candidate list”). This can lead to false-positive stale-doc findings against docs that the PR is
actively updating, violating the skill’s own constraint.

Code

internal/scaffold/fullsend-repo/skills/docs-review/SKILL.md[R127-130]

+Additionally, for each changed code file, check whether a
+documentation file covering the same feature exists based on file
+name and directory structure. If one exists and is not already in the
+candidate list, add it.

Relevance

⭐⭐ Medium

Docs-review doc tweaks often rejected (PRs #717, #2044); no prior evidence about re-excluding
PR-modified docs.

PR-#717
PR-#2044

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The instructions first exclude PR-modified docs, then add name/dir matched docs without re-applying
the “modified in PR” exclusion; the constraints section reiterates PR-modified docs must not appear
in findings.

internal/scaffold/fullsend-repo/skills/docs-review/SKILL.md[123-130]
internal/scaffold/fullsend-repo/skills/docs-review/SKILL.md[198-200]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Step 4 first excludes docs that are modified in the same PR, but the newly added “name/directory match” addition can re-introduce those excluded docs because the check is only “not already in the candidate list”.

## Issue Context
The skill later states as a constraint that PR-modified docs must not appear in findings, so step 4 should not produce candidate sets that can include PR-modified docs.

## Fix Focus Areas
- internal/scaffold/fullsend-repo/skills/docs-review/SKILL.md[123-130]

Suggested change: amend the new sentence to explicitly retain the same exclusion (e.g., “If one exists, is not modified in this PR, and is not already in the candidate list, add it.”) or state that after all candidate additions, re-filter the candidate list to remove PR-modified docs.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Related-doc match underspecified 🐞 Bug ⚙ Maintainability

Description

The new “match by file name and directory structure” instruction doesn’t define a concrete heuristic
(what directories to search, what constitutes a match, or how many candidates to add), so different
runs/agents can apply different interpretations and produce inconsistent candidate sets. This
ambiguity is duplicated into the docs-currency sub-agent instructions, increasing variance in
behavior.

Code

internal/scaffold/fullsend-repo/skills/docs-review/SKILL.md[R127-130]

+Additionally, for each changed code file, check whether a
+documentation file covering the same feature exists based on file
+name and directory structure. If one exists and is not already in the
+candidate list, add it.

Relevance

⭐⭐⭐ High

Team frequently accepts clarifying underspecified/ambiguous instructions to reduce variance (e.g.,
PR #1903).

PR-#1903

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The skill introduces the new name/dir matching step but provides no further definition elsewhere;
the same underspecified phrasing is repeated in the docs-currency sub-agent instructions.

internal/scaffold/fullsend-repo/skills/docs-review/SKILL.md[127-130]
internal/scaffold/fullsend-repo/skills/pr-review/sub-agents/docs-currency.md[18-22]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new related-doc discovery step relies on “file name and directory structure” but does not specify how to perform the mapping. Without a defined heuristic and limits, different agents may select different docs (or too many docs), reducing repeatability.

## Issue Context
This instruction is now present both in the docs-review skill (step 4) and in the docs-currency sub-agent summary, so it should be specific enough to apply consistently.

## Fix Focus Areas
- internal/scaffold/fullsend-repo/skills/docs-review/SKILL.md[127-130]
- internal/scaffold/fullsend-repo/skills/pr-review/sub-agents/docs-currency.md[18-22]

Suggested change: document an explicit heuristic (and bounds), e.g.:
- Only consider docs under known doc roots (docs/, README*, etc.).
- Match by basename/stem (case-insensitive), optionally plus one parent directory segment.
- Cap added candidates per changed file (e.g., top 1–3 matches).
- Provide 1–2 examples of expected matches/non-matches.
This makes the step reproducible and reduces over-expansion of candidates.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[qodo-code-review]

2. Related-doc match underspecified 🐞 Bug ⚙ Maintainability

The new “match by file name and directory structure” instruction doesn’t define a concrete heuristic
(what directories to search, what constitutes a match, or how many candidates to add), so different
runs/agents can apply different interpretations and produce inconsistent candidate sets. This
ambiguity is duplicated into the docs-currency sub-agent instructions, increasing variance in
behavior.

Agent Prompt

## Issue description
The new related-doc discovery step relies on “file name and directory structure” but does not specify how to perform the mapping. Without a defined heuristic and limits, different agents may select different docs (or too many docs), reducing repeatability.

## Issue Context
This instruction is now present both in the docs-review skill (step 4) and in the docs-currency sub-agent summary, so it should be specific enough to apply consistently.

## Fix Focus Areas
- internal/scaffold/fullsend-repo/skills/docs-review/SKILL.md[127-130]
- internal/scaffold/fullsend-repo/skills/pr-review/sub-agents/docs-currency.md[18-22]

Suggested change: document an explicit heuristic (and bounds), e.g.:
- Only consider docs under known doc roots (docs/, README*, etc.).
- Match by basename/stem (case-insensitive), optionally plus one parent directory segment.
- Cap added candidates per changed file (e.g., top 1–3 matches).
- Provide 1–2 examples of expected matches/non-matches.
This makes the step reproducible and reduces over-expansion of candidates.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[rh-hemartin]

Hello! We are migrating agents to an independent repository, we still need to remove files from here. Could you open this very same PR but against https://github.com/fullsend-ai/agents?

[csoceanu]

Sure, opened the same PR against the agents repo: fullsend-ai/agents#55

[fullsend-ai-retro]

🤖 Finished Retro · ✅ Success · Started 8:34 AM UTC · Completed 8:42 AM UTC
Commit: 8ac17f5 · View workflow run →

[fullsend-ai-retro]

PR #3233 was a human-authored fork PR by csoceanu modifying 2 files under internal/scaffold/fullsend-repo/ to improve the docs-review skill. It was closed without merge after a human reviewer redirected the contributor to fullsend-ai/agents (~5 hours later). The fullsend review agent was never dispatched because the contributor lacks write permission (fork PR authorization gate). This is the 4th+ documented instance of this redirection pattern. Two proposals follow: one for a CI-level deprecated-path check that covers fork PRs (gap in #3476 which only targets the review agent), and one providing evidence for #2974 (retro self-skip when no fullsend agents participated).

Proposals filed

• Add CI-level check for deprecated scaffold paths, covering fork PRs that the review agent skips (in fullsend-ai/fullsend)
• Evidence for #2974: retro dispatched on fork PR #3233 with zero fullsend agent participation (in fullsend-ai/fullsend)