crawlie v1.0.0 Crawlie.Page

Defines the struct representing a url’s state in the system.

Summary

t()

The Crawlie.Page struct type

Creates a “child page” - a new Crawlie.Page struct with depth one greate than the one of the parent and no retries

Creates a new Crawlie.Page struct from the url

Returns the Crawlie.Page object with the retry count increased

Returns the string url of the page

t()

t() :: %Crawlie.Page{depth: integer, retries: integer, uri: URI.t}

The Crawlie.Page struct type.

Fields’ meaning:

:uri - page URI
:depth - the “depth” at which the url was found while recursively crawling the pages. For example depth=0 means it was passed directly from the caller, depth=2 means the crawler followed 2 links from one of the starting urls to get to the url.
:retries - url fetch retry count. If the fetching of the url never failed before, 0.

child(page, uri)

child(Crawlie.Page.t, URI.t | String.t) :: Crawlie.Page.t

Creates a “child page” - a new Crawlie.Page struct with depth one greate than the one of the parent and no retries.

new(uri, depth \\ 0)

new(URI.t | String.t, integer) :: Crawlie.Page.t

Creates a new Crawlie.Page struct from the url

retry(this)

retry(Crawlie.Page.t) :: Crawlie.Page.t

Returns the Crawlie.Page object with the retry count increased

url(this)

url(Crawlie.Page.t) :: String.t

Returns the string url of the page