View Source treewalker_fetcher behaviour (treewalker v0.4.1)

Fetch webpages.

Link to this section Summary

Link to this section Types

-type options() :: term().
-type status_code() :: 100..599.
-type url() :: treewalker_page:url().
-type user_agent() :: treewalker_crawler_config:user_agent().

Link to this section Callbacks

-callback request(Url :: url(), UserAgent :: user_agent(), RequestOptions :: options()) ->
           {ok, {Code :: status_code(), Content :: binary()}} | {error, Reason :: term()}.

Link to this section Functions

Link to this function

request(Url, UserAgent, Options)

View Source
-spec request(url(), user_agent(), options()) -> {ok, {status_code(), binary()}} | {error, term()}.