View Source Crawler (Crawler v1.5.0)

This is the default crawl behaviour as the queue determines when an actual crawl should happen based on the available workers and the rate limit. The queue kicks off Crawler.Dispatcher.Worker which in turn calls Crawler.crawl_now/1.

crawl_now(opts)

Crawls immediately, this is used by Crawler.Dispatcher.Worker.start_link/1.

For general purpose use cases, always use Crawler.crawl/2 instead.

pause(opts)

Pauses the crawler.

resume(opts)

Resumes the crawler after it was paused.

running?(opts)

Checks whether the crawler is still crawling.

start(type, args)

Crawler is an application that gets started automatically with:

a Crawler.Store that initiates a Registry for keeping internal data

stop(opts)

Stops the crawler.

Settings View Source Crawler (Crawler v1.5.0)

Summary

Functions