Crawler
v1.5.0

View Source Crawler.Store (Crawler v1.5.0)

An internal data store for information related to each crawl.

Summary

Functions

add(arg)

Adds a URL to the registry.

add_page_data(arg, body, opts)

Adds the page data for a URL to the registry.

all_urls()
child_spec(init_arg)

Returns a specification to start this module under a supervisor.

find(arg)

Finds a stored URL and returns its page data.

find_processed(arg)

Finds a stored URL and returns its page data only if it's processed.

init(args)

Initialises a new Registry named Crawler.Store.DB.

ops_count()
ops_inc()
ops_reset()
processed(arg)

Marks a URL as processed in the registry.

start_link(opts)

Functions

Link to this function

add(arg)

View Source

Adds a URL to the registry.

Link to this function

add_page_data(arg, body, opts)

View Source

Adds the page data for a URL to the registry.

Link to this function

all_urls()

View Source
Link to this function

child_spec(init_arg)

View Source

Returns a specification to start this module under a supervisor.

See Supervisor.

Link to this function

find(arg)

View Source

Finds a stored URL and returns its page data.

Link to this function

find_processed(arg)

View Source

Finds a stored URL and returns its page data only if it's processed.

Link to this function

init(args)

View Source

Initialises a new Registry named Crawler.Store.DB.

Link to this function

ops_count()

View Source
Link to this function

ops_inc()

View Source
Link to this function

ops_reset()

View Source
Link to this function

processed(arg)

View Source

Marks a URL as processed in the registry.

Link to this function

start_link(opts)

View Source