[Feature] Add support for sitemaps

[EchedelleLR]

Would be ideal that, given a domain, the crawler was capable of detecting the sitemap or accepting a sitemap as parameter.

I would think in the following workflow:

• A url is passed -> check for sitemap -> index everything on it
• A url is passed + a no-sitemap parameter is passed -> index just that page
• A url is passed + that url is not a path itself (it goes to html or a file instead -> index just that

This would be both for main urls or secondary urls (there are sub-sitemaps and specially if you are using routes and behind reverse proxy server).

[daoudclarke]

Another option would be to allow the initial URL to be the URL of the sitemap itself.

This would be a good feature