Add robots.txt checking option

[ArkNill]

Feature Request

Currently markgrab does not check robots.txt before fetching. Add an optional respect_robots=True parameter that:

1. Fetches and parses robots.txt for the target domain
2. Checks if the URL path is allowed for the configured user agent
3. Raises RobotsDisallowed or silently skips if disallowed

This should be opt-in (default False) to maintain backward compatibility.

Motivation

Legal compliance for production deployments. Currently documented in Disclaimer but not enforced.