Stop your docs from lying. A GitHub Action that detects when code changes contradict your documentation — and offers to fix them automatically.

Every engineering team has stale docs.

A developer refactors state management from Redux to Zustand. The code ships. The ARCHITECTURE.md still says "We use Redux Toolkit for all global state." Nobody notices — until a new hire wastes two days debugging the wrong mental model.

Knowledge Diff sits in your CI and plays the role of a vigilant tech writer — one that actually reads the diff.

On every pull request, Knowledge Diff:

1. Reads the code diff — what functions, string literals, and lines were added/removed
2. Finds relevant doc sections — matches changed files/symbols against your README, ARCHITECTURE.md, CLAUDE.md, and any other docs you configure
3. Asks an LLM — "Does the code change contradict what the doc says?"
4. Comments on the PR — with specific, quote-level detail about what drifted
5. Opens a patch PR (optional) — with suggested doc updates ready for your review

🧠 Knowledge Diff — Rationale Drift Detected

🔴 src/store/cart.ts → ARCHITECTURE.md — State Management

Definite contradiction: The code replaced Redux createSlice with Zustand create(), but the doc still describes Redux as the state management solution.

Doc still says: "We use Redux Toolkit with createSlice for all global state."

Suggested update:

- We use Redux Toolkit with createSlice for all global state.
+ We use Zustand for client-side global state management.

Add this to .github/workflows/knowledge-diff.yml:

name: Knowledge Diff

on:
  pull_request:
    types: [opened, synchronize, reopened]

permissions:
  contents: write       # required for auto-patch
  pull-requests: write  # required to post comments

jobs:
  check-rationale-drift:
    runs-on: ubuntu-latest
    steps:
      - uses: oarisur/knowledge-diff@v1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}

That's it. Every PR now gets a documentation health check.

- uses: oarisur/knowledge-diff@v1
  id: drift
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    openai-api-key: ${{ secrets.OPENAI_API_KEY }}
    sensitivity: low  # only definite contradictions

- name: Fail on drift
  if: steps.drift.outputs.drift-detected == 'true'
  run: |
    echo "Definite documentation drift detected. Please update your docs."
    exit 1

- uses: oarisur/knowledge-diff@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
    llm-provider: anthropic

- uses: oarisur/knowledge-diff@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    gemini-api-key: ${{ secrets.GEMINI_API_KEY }}
    llm-provider: gemini

- uses: oarisur/knowledge-diff@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    openai-api-key: ${{ secrets.OPENAI_API_KEY }}
    auto-patch: "true"

When drift is detected, a second PR like docs/knowledge-diff-42-a1b2c3d is opened targeting the same base branch — with the suggested text replacement applied. You review and merge (or discard) at your discretion.

- uses: oarisur/knowledge-diff@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    openai-api-key: ${{ secrets.OPENAI_API_KEY }}
    doc-files: "docs/architecture/*.md,CLAUDE.md"
    sensitivity: high

PR opened / push to PR
        │
        ▼
 [Fetch PR diff]  ──►  changed code files only (by extension)
        │
        ▼
 [Fetch doc files]  ──►  README.md, ARCHITECTURE.md, docs/**/*.md
        │
        ▼
 [Keyword index]  ──►  map: symbol/path → doc sections that mention it
        │
        ▼
 [LLM comparison]  ──►  for each (code hunk, candidate doc section):
        │                 "Does this code change contradict the doc?"
        ▼
 [Drift found?]
   ├── YES ──► Post PR comment with quote-level explanation
   │           └── auto-patch: true → open a doc-fix PR
   └── NO  ──► Post "✅ all clear" comment (updates existing one)

Rather than sending entire files to the LLM (expensive, slow), Knowledge Diff:

1. Splits each doc into sections by heading
2. Builds a keyword index over all sections
3. For each changed code file, looks up the top 6 most relevant sections by keyword overlap with the changed file path and symbol names
4. Sends only those sections to the LLM

This keeps costs low and avoids irrelevant context diluting the analysis.

Add these to your workflow job:

permissions:
  contents: write       # create branches for auto-patch
  pull-requests: write  # post comments, open patch PRs

If auto-patch is false, you only need pull-requests: write.

Each PR run makes approximately N × 6 LLM calls, where N is the number of changed code files (up to max-files-per-run). Each call is a short prompt (~1,500 tokens) + a short completion (~500 tokens).

For a typical PR changing 5 files:

• ~30 calls × ~2,000 tokens ≈ ~60,000 tokens
• Cost at gpt-4o pricing: ~$0.20 per PR run

Set max-files-per-run: 10 and sensitivity: low to minimise cost on large PRs.

git clone https://github.com/oarisur/knowledge-diff
cd knowledge-diff
npm install
npm run build       # bundles to dist/index.js via ncc
npm test            # run unit tests
npm run typecheck   # verify TypeScript
npm run lint        # ESLint checks

MIT — see LICENSE.

PRs welcome. The action dogfoods itself — any change to src/ that contradicts this README.md will be caught by its own CI. 🧠