treewalker v0.4.1

API Reference treewalker v0.4.1

modules

 Modules

treewalker

This OTP application is used for crawling websites while respecting robots.txt.

treewalker_app
treewalker_crawler
treewalker_crawler_config
treewalker_crawler_sup
treewalker_crawlers_sup
treewalker_dispatcher
treewalker_fetcher
Fetch webpages.
treewalker_link_filter
Filter URL and return whether or not they must be followed.
treewalker_page
Manipulate a scrapped page.
treewalker_rate_limit
treewalker_scraper
Scrap the page for useful data and links.
treewalker_store
Store the scrapped page.
treewalker_sup
treewalker_worker