ExCrawlzy
v0.1.1

View Source ExCrawlzy (ExCrawlzy v0.1.1)

Documentation for ExCrawlzy.

Summary

Types

map_key()
post_processing()
result()
selector_tuple()

Functions

crawl(link, clients \\ [])

Request link and returns the raw content.

parse(mapping, raw_content)

Request link and returns the raw content.

Types

Link to this type

map_key()

View Source 
@type map_key() :: String.t() | atom()
Link to this type

post_processing()

View Source 
@type post_processing() :: atom() | {module(), atom()} | (any() -> String.t())
Link to this type

result()

View Source 
@type result() :: :ok | :error
Link to this type

selector_tuple()

View Source 
@type selector_tuple() :: {String.t(), post_processing()}

Functions

Link to this function

crawl(link, clients \\ [])

View Source 
@spec crawl(String.t(), [ExCrawlzy.BrowserClients.client()]) :: {result(), String.t()}

Request link and returns the raw content.

Examples 

iex> ExCrawlzy.crawl("http://some.site")
{:ok, "<!doctype html><html>  <head>    <title>the title</title>  </head>  <body>    <div id=\"the_body\">      the body      <div id=\"inner_field\">        inner field      </div>      <div id=\"inner_second_field\">        inner second field        <div id=\"the_number\">          2023        </div>      </div>      <div id=\"exist\">        this field exist      </div>      <a class=\"link_class\" href=\"http://some_external.link\"></a>      <img class=\"img_class\" src=\"http://some_external.link/image_path.jpg\" alt=\"some image\">    </div>  </body></html>"}
Link to this function

parse(mapping, raw_content)

View Source 
@spec parse(
  %{required(map_key()) => selector_tuple()},
  String.t() | Floki.html_tree() | Floki.html_node()
) :: {result(), %{required(map_key()) => String.t()}}

Request link and returns the raw content.

Examples 

iex> raw_content = "<html><head><title>the title</title></head><body><div id=\"the_body\">the body</div></body></html>"
iex> ExCrawlzy.parse(%{body: {"#the_body", :text}}, raw_content)
{:ok, %{body: "the body"}}