View Source treewalker_crawler (treewalker v0.4.1)

Link to this section Summary

Link to this section Types

-type agent_rules() :: robots:agent_rules().
-type config() :: treewalker_crawler_config:config().
-type data() :: #data{}.
-type depth() :: treewalker_crawler_config:depth().
-type dispatcher_id() :: treewalker_crawler_sup:dispatcher_id().
-type either(Left, Right) :: {ok, Left} | {error, Right}.
-type request_id() :: reference().
-type requests_by_ids() :: #{request_id() := {url(), depth()}}.
-type url() :: treewalker_page:url().

Link to this section Functions

Link to this function

handle_event(Event, Message, State, Data)

View Source
-spec start_crawler(id()) -> ok.
Link to this function

start_link(Id, DispatcherId, Config)

View Source
-spec start_link(id(), dispatcher_id(), config()) -> {ok, pid()} | {error, term()}.
-spec stop_crawler(id()) -> ok.
Link to this function

terminate(Reason, State, Data)

View Source