Crawler v1.1.1 Crawler.Store

An internal data store for information related to each crawl.

Link to this section Summary

Functions

Adds a URL to the registry.

Adds the page data for a URL to the registry.

Finds a stored URL and returns its page data.

Finds a stored URL and returns its page data only if it's processed.

Initialises a new Registry named Crawler.Store.DB.

Marks a URL as processed in the registry.

Link to this section Functions

Adds a URL to the registry.

Link to this function

add_page_data(url, body, opts)

Adds the page data for a URL to the registry.

Finds a stored URL and returns its page data.

Link to this function

find_processed(url)

Finds a stored URL and returns its page data only if it's processed.

Initialises a new Registry named Crawler.Store.DB.

Marks a URL as processed in the registry.