Module treewalker

Add and controls web crawlers.

Copyright © 2020 Antoine Gagné

Authors: Antoine Gagné (gagnantoine@gmail.com).

Description

Add and controls web crawlers.

Data Types

child()

child() = treewalker_crawlers_sup:child()

url()

url() = treewalker_page:url()

Function Index

add_crawler/2 Add a new crawler with the default configuration.
add_crawler/3 Add a new crawler with the specified configuration.
start_crawler/1 Start the specified crawler.
stop_crawler/1 Stop the specified crawler.

Function Details

add_crawler/2

add_crawler(Name::term(), Url::url()) -> {ok, child()} | {ok, child(), term()} | {error, term()}

Add a new crawler with the default configuration.

add_crawler/3

add_crawler(Name::term(), Url::url(), Custom::map()) -> {ok, child()} | {ok, child(), term()} | {error, term()}

Add a new crawler with the specified configuration.

The available options are as follow:

- scraper: Module implementing the treewalker_scraper behaviour.

- scraper_options: The options to pass to the module implementing the treewalker_scraper behaviour.

- fetcher: Module implementing the treewalker_fetcher behaviour.

- fetcher_options: The options to pass to the module implementing the treewalker_fetcher behaviour.

- max_depth: The max depth that the crawler will crawl.

start_crawler/1

start_crawler(Name::term()) -> ok

Start the specified crawler.

stop_crawler/1

stop_crawler(Name::term()) -> ok

Stop the specified crawler.


Generated by EDoc