Ethical Crawling
It is important to be polite when performing web crawling. Avoid harming scraped websites. Scraping politely:
- A polite crawler respects robots.txt.
- A polite crawler never degrades a website’s performance.
- A polite crawler identifies its creator with contact information.
- A polite crawler is not a pain in the buttocks of system administrators.