Improve HTML to markdown processor with Mozilla Readability integration

[Copilot]

Implements issue #79 by integrating Mozilla Readability into the HTML to markdown conversion pipeline to extract main content and filter out boilerplate elements like navigation, ads, and sidebars.

Changes Made

Enhanced src/lib/utils/html.ts:

• Added extractMainContent() function that uses Mozilla Readability to clean HTML before conversion
• Integrated Readability processing into the existing html2markdown() function
• Maintains full backward compatibility with graceful fallback to original HTML if Readability fails
• Preserves the existing unified processor + TurndownService conversion pipeline

Dependencies Added:

• @mozilla/readability - Mozilla's content extraction library
• jsdom - DOM environment for server-side Readability processing
• @types/jsdom and @types/domhandler for TypeScript support

Benefits

The improved processor now provides cleaner markdown output by:

• Extracting main article content from complex web pages
• Filtering out navigation menus, advertisements, and footer content
• Preserving essential content like headings, paragraphs, code blocks, and lists
• Maintaining existing functionality for simple HTML content

Testing

Validated on PyPI project pages (e.g., /pypi/requests) showing significant improvement in markdown quality:

• Main content properly extracted and formatted
• Boilerplate content successfully filtered out
• Code examples and installation instructions cleanly preserved
• All builds and linting pass successfully

This enhancement is particularly beneficial for the PyPI project description conversion feature, resulting in much cleaner and more focused markdown output.

---

💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

[bolt-new-by-stackblitz]

Run & review this pull request in StackBlitz Codeflow.

[codesandbox]

Review or Edit in CodeSandbox

Open the branch in Web Editor • VS Code • Insiders

Open Preview