View Source Crawler.Fetcher.Recorder (Crawler v1.5.0)
Records information about each crawl for internal use.
Summary
Functions
Stores page data in Crawler.Store.DB for internal or external consumption, if enabled.
Records information about each crawl for internal use.
Functions
Stores page data in Crawler.Store.DB for internal or external consumption, if enabled.
Examples
iex> Recorder.maybe_store_page("body", %{store: nil})
{:ok, nil}
iex> Recorder.record(url: "url", depth: 2)
iex> Recorder.maybe_store_page("body", %{store: Store, url: "url", scope: nil})
{:ok, {%Page{url: "url", body: "body", opts: %{store: Store, url: "url", scope: nil}}, %Page{url: "url", body: nil}}}Records information about each crawl for internal use.
Examples
iex> Recorder.record(url: "url1", depth: 2)
{:ok, %{depth: 3, url: "url1"}}
iex> Recorder.record(url: "url2", depth: 2)
iex> Store.find({"url2", nil})
%Page{url: "url2"}