Crawly.Middlewares.RobotsTxt (Crawly v0.16.0) View Source
A robots.txt file tells search engine crawlers which pages or files the crawler can or can't request from your site. This is used mainly to avoid overloading a site with requests!
No options are required for this middleware. Any tuple-based configurations options passed will be ignored.