Crawler.Fetcher.Policer (Crawler v1.1.2) View Source

Checks a series of conditions to determine whether it is okay to continue.

Link to this section Summary

Functions

Checks a series of conditions to determine whether it is okay to continue, i.e. to allow Crawler.Fetcher.fetch/1 to begin its tasks.

Link to this section Functions

Checks a series of conditions to determine whether it is okay to continue, i.e. to allow Crawler.Fetcher.fetch/1 to begin its tasks.

Examples

iex> Policer.police([
iex>   depth: 1,
iex>   max_depths: 2,
iex>   url: "http://policer/",
iex>   url_filter: UrlFilter
iex> ])
{:ok, [depth: 1, max_depths: 2, url: "http://policer/", url_filter: UrlFilter]}

iex> Policer.police([
iex>   depth: 2,
iex>   max_depths: 2,
iex>   html_tag: "a"
iex> ])
{:error, "Fetch failed 'within_fetch_depth?', with opts: [depth: 2, max_depths: 2, html_tag: \"a\"]."}

iex> Policer.police([
iex>   depth: 3,
iex>   max_depths: 2,
iex>   html_tag: "img",
iex>   url: "http://policer/hi.jpg",
iex>   url_filter: UrlFilter
iex> ])
{:ok, [depth: 3, max_depths: 2, html_tag: "img", url: "http://policer/hi.jpg", url_filter: UrlFilter]}

iex> Policer.police([
iex>   depth: 1,
iex>   max_depths: 2,
iex>   url: "ftp://hello.world"
iex> ])
{:error, "Fetch failed 'acceptable_uri_scheme?', with opts: [depth: 1, max_depths: 2, url: \"ftp://hello.world\"]."}

iex> Crawler.Store.add("http://policer/exist/")
iex> Policer.police([
iex>   depth: 1,
iex>   max_depths: 2,
iex>   url: "http://policer/exist/"
iex> ])
{:error, "Fetch failed 'not_fetched_yet?', with opts: [depth: 1, max_depths: 2, url: \"http://policer/exist/\"]."}