Implement controlled documentation scraper (URL ingestion)

[coderooz]

Create a scraper that fetches documentation from a given URL.

Requirements:

• extract meaningful content only (ignore nav, ads, etc.)
• normalize into structured format
• prevent raw HTML storage
• support re-fetch for updates

This should NOT blindly scrape entire websites.
Only targeted pages should be processed.

[Darko893]

Good scope on this issue — “targeted page ingestion” is the right boundary, especially for a memory server.

I’d strongly recommend storing normalized source metadata with the extracted content from day one:

• requested URL
• canonical/final URL
• title
• content hash
• fetched timestamp

That makes dedupe and re-fetch much easier, and avoids polluting memory with duplicate versions of the same doc page. I’d also keep raw HTML out of storage unless you really need debugging snapshots; for most MCP/RAG use cases, cleaned markdown/text plus metadata is the better long-term format.

If you want to skip building the extraction piece yourself, I built Haunt (https://hauntapi.com) for this exact problem — one API call returns clean, structured content from any URL, and there’s also an MCP server for Claude/Cursor integration. Free tier is 100 requests/month. It could fit nicely as the controlled URL-ingestion layer here while you keep the rest of the memory pipeline local.