Ethical Crawling

It is important to be polite when performing web crawling. Avoid harming scraped websites. Scraping politely:

  1. A polite crawler respects robots.txt.
  2. A polite crawler never degrades a website’s performance.
  3. A polite crawler identifies its creator with contact information.
  4. A polite crawler is not a pain in the buttocks of system administrators.