Crawler (Crawler v1.1.2) View Source

A high performance web crawler in Elixir.

Link to this section Summary

Functions

Crawls immediately, this is used by Crawler.Dispatcher.Worker.start_link/1.

Pauses the crawler.

Resumes the crawler after it was paused.

Crawler is an application that gets started automatically with

Stops the crawler.

Link to this section Functions

Enqueues a crawl, via Crawler.QueueHandler.enqueue/1.

This is the default crawl behaviour as the queue determines when an actual crawl should happen based on the available workers and the rate limit. The queue kicks off Crawler.Dispatcher.Worker which in turn calls Crawler.crawl_now/1.

Crawls immediately, this is used by Crawler.Dispatcher.Worker.start_link/1.

For general purpose use cases, always use Crawler.crawl/2 instead.

Pauses the crawler.

Resumes the crawler after it was paused.

Crawler is an application that gets started automatically with:

Stops the crawler.