Crawler v1.1.2

Crawler.Dispatcher (Crawler v1.1.2) View Source

Dispatches requests to a queue for crawling.

Link to this section Summary

Functions

dispatch(request, opts)

Takes the request argument which is a tuple containing either

Link to this section Functions

Link to this function

dispatch(request, opts)

View Source

Takes the request argument which is a tuple containing either:

  • {_, link, _, url} when it's a link that got transformed into a URL
  • {_, url} when it's a URL already

And issues Crawler.crawl/2 to initiate the crawl.