Skip to content

Add Agent-Friendly Docs rules, skill, and CI - #2944

Closed
shahbaz17 wants to merge 8 commits into
mainfrom
feat-afdocs-skill-ci
Closed

Add Agent-Friendly Docs rules, skill, and CI#2944
shahbaz17 wants to merge 8 commits into
mainfrom
feat-afdocs-skill-ci

Conversation

@shahbaz17

@shahbaz17 shahbaz17 commented Jun 3, 2026

Copy link
Copy Markdown
Member

Description

Introduce Agent-Friendly Docs enforcement: add .cursor/rules/agent-friendly-docs.mdc (AFDocs conventions and checks), .cursor/skills/agent-friendly-docs/SKILL.md (PR review checklist/skill for reviewers), and a GitHub Actions workflow .github/workflows/afdocs-check.yml. The workflow enforces llms.txt size, .md parity, and alternate-link injection at build-time (Tier 1 via scripts/verify-llms-output.js) and runs an advisory afdocs check against Vercel preview deployments (Tier 2). These additions codify AFDocs/SEO/orphan-route rules and provide automated reviewer guidance to prevent regressions in agent discoverability.

Issue(s) fixed

Fixes #

Preview

Checklist

  • If this PR updates or adds documentation content that changes or adds technical meaning, it has received an approval from an engineer or DevRel from the relevant team.
  • If this PR updates or adds documentation content, it has received an approval from a technical writer.

External contributor checklist

  • I've read the contribution guidelines.
  • I've created a new issue (or assigned myself to an existing issue) describing what this PR addresses.

Note

Low Risk
Changes are limited to Cursor docs, CI, and the verify script; they do not alter runtime site behavior, though Tier 1 can block PRs if build output fails the new invariants.

Overview
Adds Agent-Friendly Docs (AFDocs) guardrails for authors and reviewers, plus two-tier CI so agent/SEO invariants are enforced without changing the live docs build.

Authoring & review: A new always-on Cursor rule (.cursor/rules/agent-friendly-docs.mdc) documents invariants for llms.txt, markdown parity, routing/orphans, and chain-page URL formatting. A matching review skill (.cursor/skills/agent-friendly-docs/SKILL.md) gives reviewers a structured checklist aligned with prior AFDocs work (#2926/#2929).

CI: .github/workflows/afdocs-check.yml runs Tier 1 on every PR to main: npm run build then node scripts/verify-llms-output.js as a blocking gate. Tier 2 runs on successful Vercel Preview deployment_status, executing pinned afdocs@0.4.0 from an isolated .github/afdocs/ install (Node 22, lockfile, no npx) and upserting an AFDocs scorecard PR comment; Tier 2 is advisory (continue-on-error) until the CLI stabilizes.

Verification script: scripts/verify-llms-output.js is expanded from a dev sanity check into the advertised hard gate: Part 1 still regenerates LLM artifacts from source; Part 2 walks build/ for root llms.txt size, per-page .md siblings, and injector markers (data-llms-md-alt, data-llms-directive), exiting non-zero on violations (skipping Part 2 locally if build/ is missing). .gitignore ignores .github/afdocs/node_modules and .tmp-*.

Reviewed by Cursor Bugbot for commit 2bc30a7. Bugbot is set up for automated code reviews on this repo. Configure here.

Introduce Agent-Friendly Docs enforcement: add .cursor/rules/agent-friendly-docs.mdc (AFDocs conventions and checks), .cursor/skills/agent-friendly-docs/SKILL.md (PR review checklist/skill for reviewers), and a GitHub Actions workflow .github/workflows/afdocs-check.yml. The workflow enforces llms.txt size, .md parity, and alternate-link injection at build-time (Tier 1 via scripts/verify-llms-output.js) and runs an advisory afdocs check against Vercel preview deployments (Tier 2). These additions codify AFDocs/SEO/orphan-route rules and provide automated reviewer guidance to prevent regressions in agent discoverability.
@shahbaz17
shahbaz17 requested review from a team as code owners June 3, 2026 12:23
@vercel

vercel Bot commented Jun 3, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
metamask-docs Ready Ready Preview, Comment Jun 3, 2026 3:27pm

Request Review

Comment thread .github/workflows/afdocs-check.yml
Comment thread .github/workflows/afdocs-check.yml Outdated
Comment thread .github/workflows/afdocs-check.yml Fixed
@github-actions

github-actions Bot commented Jun 3, 2026

Copy link
Copy Markdown

AFDocs Agent Score (preview)

{
  "total": 21,
  "pass": 10,
  "warn": 1,
  "fail": 3,
  "skip": 7,
  "error": 0
}

Introduce a dedicated, lockfile-pinned AFDocs install under .github/afdocs and update CI to install and invoke the local binary (avoiding npx/unpinned installs). Update the afdocs-check workflow to pin Node 22 for the advisory job, run the pinned runner, and treat the verify script as a hard gate for build-time invariants. Enhance scripts/verify-llms-output.js to a two-part verifier: a generator sanity check (copies/validates static/llms.txt) and a production build gate that asserts llms.txt size (50k char threshold), per-page .md parity, and alternate-link/body directive injection; failures produce non-zero exit. Also add .github/afdocs/package.json (and lockfile), ignore the runner node_modules in .gitignore, and include small .tmp-fakebuild fixtures used by the verifier.
@socket-security

socket-security Bot commented Jun 3, 2026

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedafdocs@​0.4.07710010093100

View full report

@socket-security

socket-security Bot commented Jun 3, 2026

Copy link
Copy Markdown

Caution

MetaMask internal reviewing guidelines:

  • Do not ignore-all
  • Each alert has instructions on how to review if you don't know what it means. If lost, ask your Security Liaison or the supply-chain group
  • Copy-paste ignore lines for specific packages or a group of one kind with a note on what research you did to deem it safe.
    @SocketSecurity ignore npm/PACKAGE@VERSION
Action Severity Alert  (click "▶" to expand/collapse)
Block Medium
Network access: npm afdocs in module globalThis["fetch"]

Module: globalThis["fetch"]

Location: Package overview

From: .github/afdocs/package-lock.jsonnpm/afdocs@0.4.0

ℹ Read more on: This package | This alert | What is network access?

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: Packages should remove all network access that is functionally unnecessary. Consumers should audit network access to ensure legitimate use.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/afdocs@0.4.0. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn Low
Potential code anomaly (AI signal): npm yaml is 100.0% likely to have a medium risk anomaly

Notes: The code presents a standard, non-malicious NodeBase component used in YAML/JS conversion. The primary risk surface is the optional reviver and onAnchor callbacks provided by the user: if untrusted, these can execute arbitrary code or influence the transformed representation via applyReviver or the reviver itself. This is expected behavior for extensible YAML libraries; ensure callbacks come from trusted sources and sandbox or validate revivers where possible.

Confidence: 1.00

Severity: 0.60

From: .github/afdocs/package-lock.jsonnpm/afdocs@0.4.0npm/yaml@2.9.0

ℹ Read more on: This package | This alert | What is an AI-detected potential code anomaly?

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: An AI system found a low-risk anomaly in this package. It may still be fine to use, but you should check that it is safe before proceeding.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/yaml@2.9.0. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

View full report

Comment thread .github/workflows/afdocs-check.yml
@github-actions

github-actions Bot commented Jun 3, 2026

Copy link
Copy Markdown

AFDocs Agent Score (preview)

{
  "total": 21,
  "pass": 10,
  "warn": 1,
  "fail": 3,
  "skip": 7,
  "error": 0
}

Prefer github.event.deployment_status.environment_url when determining the preview URL for the AFDocs check, falling back to target_url only if environment_url is empty. This avoids pointing AFDocs at Vercel's log/target_url (which GitHub may map to log_url), passes the values via environment variables to prevent shell injection, and exits with a clear error if no preview URL is available.
Comment thread .tmp-fakebuild/llms.txt Outdated
Comment thread scripts/verify-llms-output.js Outdated
@github-actions

github-actions Bot commented Jun 3, 2026

Copy link
Copy Markdown

AFDocs Agent Score (preview)

{
  "total": 21,
  "pass": 10,
  "warn": 1,
  "fail": 3,
  "skip": 7,
  "error": 0
}

shahbaz17 added 2 commits June 3, 2026 19:46
Add /.tmp-* to .gitignore and remove throwaway .tmp-fakebuild fixtures. Update scripts/verify-llms-output.js: rename articlePages to contentPages, update console output, and change the page detection to validate pages that use <article> or fall back to <main>, with updated comments to mirror the injector's html-to-markdown fallback.
@github-actions

github-actions Bot commented Jun 3, 2026

Copy link
Copy Markdown

AFDocs Agent Score (preview)

{
  "total": 21,
  "pass": 10,
  "warn": 1,
  "fail": 3,
  "skip": 7,
  "error": 0
}

1 similar comment
@github-actions

github-actions Bot commented Jun 3, 2026

Copy link
Copy Markdown

AFDocs Agent Score (preview)

{
  "total": 21,
  "pass": 10,
  "warn": 1,
  "fail": 3,
  "skip": 7,
  "error": 0
}

shahbaz17 added 2 commits June 3, 2026 20:08
Improve the AFDocs GitHub Actions reporting by producing a richer, upsertable scorecard comment. Adds a hidden marker so the workflow updates a single comment instead of posting new ones, surfaces the pinned afdocs CLI version, and uses icons and formatted text for the summary. Parses afdocs-report.json more robustly and links to the job log on parse failure. Includes a details table of failing/warning/error checks (omitting skips) and reuses an existing comment when present (lists comments with per_page=100 and updates the matching comment), returning early if no PR is associated with the deployment.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 2bc30a7. Configure here.

exit 1
fi
echo "Checking $PREVIEW_URL"
.github/afdocs/node_modules/.bin/afdocs check "$PREVIEW_URL" --format json --verbose | tee afdocs-report.json

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pipe masks afdocs exit code

Medium Severity

The Tier 2 step runs afdocs check through a pipe into tee without pipefail, so the step’s exit status comes from tee, not afdocs. When the CLI exits non-zero, the Actions step can still succeed, hiding failures until someone reads the PR comment or log.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 2bc30a7. Configure here.

@github-actions

github-actions Bot commented Jun 3, 2026

Copy link
Copy Markdown

AFDocs Agent Score (preview)

Checked https://metamask-docs-4hkeuvnej-consensys-ddffed67.vercel.app with afdocs@0.4.0.

❌ 10/21 checks passed — ✅ 10 passed · ⚠️ 1 warning · ❌ 3 failed · ⚪ 7 skipped

Checks needing attention
Status Check Message
markdown-url-support No pages support .md URLs (0/0 tested)
content-negotiation Server ignores Accept: text/markdown header (0/0 pages return markdown)
page-size-html 13 of 25 pages convert to over 100K chars (max 2451K, 0% boilerplate)
⚠️ llms-txt-directive llms.txt directive found in 1 of 25 pages (24 missing)

Run against the Vercel preview deployment. Skipped checks are unimplemented in afdocs@0.4.0 (not doc problems). Full output in the job log.

@shahbaz17
shahbaz17 marked this pull request as draft June 3, 2026 14:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants