View Source Crawler.Fetcher (Crawler v1.5.0)
Fetches pages and perform tasks on them.
Summary
Functions
Fetches a URL by
Functions
Fetches a URL by:
- verifying whether the URL needs fetching through
Crawler.Fetcher.Policer.police/1 - recording data for internal use through
Crawler.Fetcher.Recorder.record/1 - fetching the URL
- performing retries upon failed fetches through
Crawler.Fetcher.Retrier.perform/2