0-scraper

Web scraper

Description

This is a simple web scraper used to scrape text content from a website

Installation

npm install

Usage

• List URLs you want to scrape into config.json.
• Run the scraper with node scraper.js

Special Bonus Easter Egg

The file clean-sitemap.js can be used to convert sitemap.xml files to a list of URLs in a .txt file 🐣

• Put a sitemap.xml in the project root director and run node clean-sitemap.js

Icon credits

scraper-open-graph-image.png by Dby Freepik

License

MIT.