Robots.txt parser with caching. Modelled after Kryten

Caches the robots.txt files from different hosts in memory

In charge of fetching the actual robots.txt files

Represents one host's robots.txt files

Parses a robots.txt file