Markdown Link Checker CLI

[jamesrxian]

Markdown Link Checker CLI

Build a markdown link checker CLI that reads a markdown file, extracts all URLs, checks them with HTTP HEAD requests, and reports broken links.

Status

• gathering requirements

• research

• design

• plan

• spec-ready

Plan

• Step 1: Project setup and data models
• Step 2: Markdown link parser
• Step 3: Link checker, robots.txt filter, and rate limiter
• Step 4: Runner, reporter, and CLI entry point
• Step 5: Integration tests

[jamesrxian]

[bot reply] 👋 Welcome! I'll guide you through a spec generation process on this issue.

Where would you like to start?

1. Requirements clarification (recommended) — refine the idea through Q&A
2. Research first — investigate specific technical topics before refining requirements

Reply with your choice (1 or 2).

[jamesrxian]

1

[jamesrxian]

[bot reply] Great, let's clarify the requirements! I'll ask one question at a time.

Q1: What programming language should this CLI be built in? For example: Python, Node.js/TypeScript, Go, Rust, etc.

[jamesrxian]

Python 3.10+. Use the requests library for HTTP HEAD checks and argparse for CLI argument parsing.

[jamesrxian]

[bot reply] Q2: What types of links should be extracted and checked? For example:

• Inline links [text](url) only?
• Also reference-style links [text][ref] with [ref]: url definitions?
• Auto-linked bare URLs (e.g., https://example.com)?
• Image links ![alt](url)?
• HTML <a href="..."> tags embedded in the markdown?

[jamesrxian]

Check all markdown link types: inline links, reference-style links, and image links. Skip bare URLs and HTML anchor tags for now.

[jamesrxian]

[bot reply] Q3: What output format should the CLI use? For example:

• Plain text report to stdout (e.g., one line per broken link with file, line number, URL, status code)?
• JSON output option for machine consumption?
• Color-coded terminal output?
• Exit code conventions (e.g., exit 0 if all links are OK, exit 1 if any are broken)?

[jamesrxian]

Plain text to stdout with one line per broken link showing file path, line number, URL, and HTTP status code. Support a --json flag for JSON output. Exit 0 for all OK, exit 1 for broken links found. That's enough questions — let's move on to requirements complete.

[jamesrxian]

requirements.md

Overview

Build a markdown link checker CLI that reads a markdown file, extracts all URLs, checks them with HTTP HEAD requests, and reports broken links.

Requirements Q&A

Q1: What programming language should this CLI be built in?
A1: Python 3.10+. Use the requests library for HTTP HEAD checks and argparse for CLI argument parsing.

Q2: What types of links should be extracted and checked?
A2: Check all markdown link types: inline links [text](url), reference-style links [text][ref] with [ref]: url definitions, and image links ![alt](url). Skip bare URLs and HTML anchor tags for now.

Q3: What output format should the CLI use?
A3: Plain text to stdout with one line per broken link showing file path, line number, URL, and HTTP status code. Support a --json flag for JSON output. Exit 0 for all OK, exit 1 for broken links found.

Consolidated Requirements

1. Language & Stack: Python 3.10+, requests for HTTP, argparse for CLI
2. Input: One or more markdown file paths as CLI arguments
3. Link Extraction:
   • Inline links: [text](url)
   • Reference-style links: [text][ref] with [ref]: url
   • Image links: ![alt](url)
   • Skip: bare URLs, HTML <a> tags
4. Link Checking: HTTP HEAD requests to each extracted URL
5. Output (plain text, default): One line per broken link: <file>:<line> <url> <status_code>
6. Output (JSON, --json flag): JSON array of objects with file, line, url, status_code fields
7. Exit codes: 0 = all links OK, 1 = broken links found

[jamesrxian]

[bot reply] ✅ Requirements gathered and posted above. What would you like to do next?

1. Review progress (checkpoint) — review the requirements before moving to design
2. Fast forward — generate design, plan, and e2e in one shot (no intermediate approvals)
3. Research first — investigate specific technical topics before designing

Reply with your choice (1, 2, or 3).

[jamesrxian]

1

[jamesrxian]

[bot reply] ## 📋 Progress Checkpoint

Requirements — Key Decisions

• Language & Stack: Python 3.10+, requests for HTTP, argparse for CLI
• Link Types: Inline [text](url), reference-style [text][ref], image ![alt](url). Skip bare URLs and HTML <a> tags.
• Output: Plain text (default) — <file>:<line> <url> <status_code>. JSON via --json flag.
• Exit Codes: 0 = all OK, 1 = broken links found.

Research Findings

• No research conducted yet.

Gaps & Open Questions

• Timeout/retries: No timeout or retry policy defined for HTTP requests. Should there be a --timeout flag? How to handle connection errors vs. HTTP error codes?
• Concurrency: Should links be checked sequentially or concurrently (e.g., with concurrent.futures)?
• Scope: Single file only, or accept multiple files / glob patterns / directory recursion?
• Fragment handling: Should #fragment anchors be validated (e.g., checking heading IDs in local files)?
• Local links: Should relative file links (e.g., ./other.md) be checked for existence?
• Rate limiting: Any throttling between requests to avoid being blocked?

Readiness Assessment

The core requirements are clear enough to proceed to design. The gaps above are reasonable defaults that can be decided during design. However, if you'd like to nail down any of these questions first, we can go back to requirements or do research.

---

What would you like to do next?

1. Proceed to design — start designing the architecture and components
2. Back to requirements — clarify the open questions above
3. More research — investigate technical topics (e.g., markdown parsing libraries, concurrency patterns)

Reply with your choice (1, 2, or 3).

[jamesrxian]

proceed to design

[jamesrxian]

design.md

1. Overview

mdlinkcheck is a Python CLI tool that reads one or more markdown files, extracts all URLs from inline links, reference-style links, and image links, validates each URL with an HTTP HEAD request, and reports broken links. It supports plain text and JSON output formats, making it suitable for both interactive use and CI pipelines.

2. Goal & Constraints

Goal: Provide a fast, reliable command-line tool to detect broken URLs in markdown documents, with clear output and proper exit codes for CI integration.

Constraints:

• Python 3.10+ (use modern syntax: match, | union types)
• Only external dependencies: requests (HTTP) — no markdown parsing libraries
• CLI via argparse — no third-party CLI frameworks
• Must handle network errors gracefully (timeouts, DNS failures, connection refused)
• Must not overwhelm target servers — limit to 5 concurrent requests per domain
• Must respect robots.txt — skip URLs disallowed for the checker's User-Agent
• External dependencies: requests (HTTP), urllib.robotparser (stdlib, robots.txt parsing)
• Single-file module structure for simplicity (can be split later if needed)

3. Architecture Overview

flowchart TD
    CLI[CLI Entry Point<br/>argparse] --> Runner[Runner<br/>orchestrates per-file checking]
    Runner --> Parser[Markdown Link Parser<br/>regex-based extraction]
    Runner --> RobotsFilter[Robots.txt Filter<br/>urllib.robotparser]
    Runner --> RateLimiter[Rate Limiter<br/>per-domain semaphore]
    Runner --> Checker[Link Checker<br/>HTTP HEAD requests]
    Runner --> Reporter[Reporter<br/>formats output]

    Parser -->|List of Link objects| Runner
    RobotsFilter -->|filtered URLs| Runner
    RateLimiter -->|throttled access| Checker
    Checker -->|List of CheckResult objects| Runner
    Runner -->|List of CheckResult objects| Reporter
    Reporter -->|stdout| Output[Plain Text or JSON]

    CLI -->|exit code 0 or 1| Exit[Process Exit]

Loading

Data flow: CLI parses args → Runner iterates over input files → Parser extracts links from each file → Robots.txt filter removes disallowed URLs → Rate limiter throttles per-domain concurrency (max 5) → Checker validates each unique URL → Reporter formats results → CLI sets exit code.

4. Components & Interfaces

4.1 CLI Entry Point (cli.py / __main__.py)

Responsibility: Parse command-line arguments, invoke the runner, handle exit codes.

def main(argv: list[str] | None = None) -> int:
    """Entry point. Returns exit code (0 = OK, 1 = broken links, 2 = error)."""

Arguments:

Argument	Type	Description
files	positional, nargs='+'	One or more markdown file paths
--json	flag	Output results as JSON
--timeout	int, default=10	HTTP request timeout in seconds

4.2 Markdown Link Parser (parser.py)

Responsibility: Extract all links from a markdown file's content, with line numbers.

@dataclass
class Link:
    url: str
    line_number: int
    link_text: str
    link_type: str  # "inline", "reference", "image"

def extract_links(content: str) -> list[Link]:
    """Extract all markdown links from content string."""

Regex patterns:

• Inline links: \[([^\]]*)\]\(([^)]+)\) — but must exclude image links (no preceding !)
• Image links: !\[([^\]]*)\]\(([^)]+)\)
• Reference definitions: ^\[([^\]]+)\]:\s+(.+)$ (multiline)
• Reference usages: \[([^\]]*)\]\[([^\]]*)\] — resolved against definitions

Edge cases:

• Nested brackets in link text
• URLs with parentheses (e.g., Wikipedia links)
• Fragment-only links (#section) — skip these (not external URLs)
• Mailto links (mailto:...) — skip these
• Duplicate URLs across lines — deduplicate for checking, but report all occurrences

4.3 Link Checker (checker.py)

Responsibility: Validate URLs via HTTP HEAD requests.

@dataclass
class CheckResult:
    url: str
    status_code: int | None  # None if connection failed
    error: str | None        # Error message if connection failed
    ok: bool                 # True if 2xx status

def check_url(url: str, timeout: int = 10) -> CheckResult:
    """Check a single URL with HTTP HEAD. Falls back to GET if HEAD returns 405."""

def check_urls(urls: list[str], timeout: int = 10) -> dict[str, CheckResult]:
    """Check multiple URLs. Deduplicates and returns url -> result mapping."""

Behavior:

• Send HTTP HEAD request first
• If HEAD returns 405 (Method Not Allowed), retry with GET
• Treat 2xx as OK, everything else as broken
• On connection error / timeout: mark as broken with error message
• Follow redirects (requests default behavior)
• Set a reasonable User-Agent header (mdlinkcheck/1.0) to avoid bot-blocking
• All requests go through the rate limiter (max 5 concurrent per domain)

4.3.1 Robots.txt Filter (robots.py)

Responsibility: Fetch and cache robots.txt per domain, filter out disallowed URLs.

class RobotsFilter:
    """Fetches and caches robots.txt per domain, checks URL access."""

    def __init__(self, user_agent: str = "mdlinkcheck", timeout: int = 10):
        ...

    def is_allowed(self, url: str) -> bool:
        """Check if the URL is allowed by the domain's robots.txt.
        Fetches and caches robots.txt on first access per domain.
        If robots.txt cannot be fetched, defaults to allowed."""

    def filter_urls(self, urls: list[str]) -> tuple[list[str], list[str]]:
        """Split URLs into (allowed, disallowed) lists."""

Behavior:

• Uses urllib.robotparser.RobotFileParser from stdlib
• Caches parsed robots.txt per domain (scheme + netloc) for the duration of the run
• If robots.txt fetch fails (404, timeout, etc.), assume all URLs are allowed
• User-Agent for robots.txt matching: mdlinkcheck
• Disallowed URLs are reported in output as "skipped (robots.txt)" rather than "broken"

4.3.2 Rate Limiter (ratelimiter.py)

Responsibility: Limit concurrent requests to 5 per domain using concurrent.futures.ThreadPoolExecutor with per-domain semaphores.

class DomainRateLimiter:
    """Limits concurrent requests per domain."""

    def __init__(self, max_per_domain: int = 5):
        ...

    def check_urls(
        self, urls: list[str], check_fn: Callable[[str], CheckResult], timeout: int = 10
    ) -> dict[str, CheckResult]:
        """Check URLs concurrently, limiting to max_per_domain concurrent requests per domain."""

Behavior:

• Groups URLs by domain (scheme + netloc)
• Uses threading.Semaphore(5) per domain to limit concurrency
• Uses concurrent.futures.ThreadPoolExecutor for parallel execution across domains
• Total thread pool size: min(32, len(urls)) to avoid excessive threads

4.4 Reporter (reporter.py)

Responsibility: Format check results for output.

@dataclass
class BrokenLink:
    file: str
    line: int
    url: str
    status_code: int | None
    error: str | None

def format_text(broken_links: list[BrokenLink]) -> str:
    """Format as plain text: one line per broken link."""

def format_json(broken_links: list[BrokenLink]) -> str:
    """Format as JSON array."""

Plain text format:

path/to/file.md:42 https://example.com/broken 404
path/to/file.md:87 https://down.example.com (connection error: Connection refused)

JSON format:

[
  {"file": "path/to/file.md", "line": 42, "url": "https://example.com/broken", "status_code": 404, "error": null},
  {"file": "path/to/file.md", "line": 87, "url": "https://down.example.com", "status_code": null, "error": "Connection refused"}
]

4.5 Runner (runner.py)

Responsibility: Orchestrate the full pipeline for one or more files.

def run(files: list[str], timeout: int = 10, max_per_domain: int = 5) -> list[BrokenLink]:
    """Run link checking on all files. Returns list of broken links."""

Algorithm:

1. For each file: read content, extract links via parser
2. Collect all unique URLs across all files
3. Filter URLs through robots.txt checker — disallowed URLs are marked as "skipped"
4. Check allowed URLs via rate limiter (max 5 concurrent per domain)
5. Map results back to all file/line occurrences
6. Return sorted list of BrokenLink objects (sorted by file, then line)

5. Data Models

classDiagram
    class Link {
        +str url
        +int line_number
        +str link_text
        +str link_type
    }

    class CheckResult {
        +str url
        +int|None status_code
        +str|None error
        +bool ok
    }

    class BrokenLink {
        +str file
        +int line
        +str url
        +int|None status_code
        +str|None error
    }

    class SkippedLink {
        +str url
        +str reason
    }

    Link --> SkippedLink : filtered by robots.txt
    Link --> CheckResult : checked via rate limiter
    CheckResult --> BrokenLink : mapped to file context

Loading

6. Error Handling

Failure Mode	Behavior
File not found	Print error to stderr, continue with other files, exit 2
File read error (permissions)	Print error to stderr, continue, exit 2
DNS resolution failure	Mark link as broken with error message
Connection timeout	Mark link as broken with error "Connection timed out"
Connection refused	Mark link as broken with error "Connection refused"
SSL certificate error	Mark link as broken with error message
HTTP 4xx/5xx	Mark link as broken with status code
HTTP 405 on HEAD	Retry with GET request
Redirect loop	requests raises TooManyRedirects — mark as broken
robots.txt fetch error	Assume all URLs for that domain are allowed
robots.txt disallows URL	Skip URL, report as "skipped (robots.txt)"
No files provided	argparse prints usage, exits 2

Exit code logic:

• 0: All links in all files are OK (or no links found)
• 1: At least one broken link found
• 2: Input error (file not found, permissions, no files given)

7. Integration Testing

Tests that verify cross-component behavior:

1. Parser → Checker integration: Parse a markdown string with known links, check them, verify results map correctly back to line numbers
2. Full pipeline with mock HTTP: Use responses or unittest.mock to mock requests, run the full pipeline from file to output, verify plain text and JSON output formats
3. Multiple files with shared URLs: Verify URL deduplication — a URL appearing in 3 files should only be checked once, but reported for all occurrences
4. HEAD-to-GET fallback: Mock a server that returns 405 on HEAD, verify the checker retries with GET
5. Mixed success/failure: File with both valid and broken links produces correct output and exit code 1
6. Robots.txt filtering: Mock robots.txt that disallows certain paths, verify those URLs are skipped and reported as "skipped (robots.txt)"
7. Rate limiting: Check 10+ URLs on the same domain, verify no more than 5 concurrent requests via mock timing

8. E2E Testing

Key end-to-end scenarios to verify:

1. Happy path: Run on a markdown file where all links are valid → exit 0, no output
2. Broken links detected: Run on a file with known broken URLs → exit 1, broken links listed
3. JSON output: Run with --json flag → valid JSON array on stdout
4. Multiple files: Run on two files → results from both files in output
5. Non-existent file: Pass a file that doesn't exist → error message on stderr, exit 2
6. Reference-style links: File using reference-style links → links correctly extracted and checked
7. Image links: File with image links → image URLs checked
8. Timeout handling: URL that times out → reported as broken with error message

9. Appendices

Technology Choices

Choice	Rationale
requests over urllib3/httpx	User requirement; widely used, simple API
argparse over click/typer	User requirement; stdlib, no extra deps
Regex over markdown parser	Simpler for link extraction only; avoids heavy deps like markdown-it-py
concurrent.futures for concurrency	Stdlib; enables per-domain rate limiting with semaphores
urllib.robotparser for robots.txt	Stdlib; no extra dependency needed

Future Enhancements (out of scope)

• Directory recursion / glob patterns
• Local/relative link validation
• Fragment anchor validation
• Caching of results across runs
• Config file support (.mdlinkcheck.yaml)

[jamesrxian]

[bot reply] 📐 Design document posted above. Please review and let me know:

1. Approve — looks good, proceed to implementation plan
2. Request changes — describe what needs to be updated
3. Gaps found — need to go back to requirements to clarify scope

Reply with your choice (1, 2, or 3), plus any feedback.