Crawly.Manager (Crawly v0.17.2) View Source

Crawler manager module

This module is responsible for spawning all processes related to a given Crawler.

The manager spawns the following processes tree. ┌────────────────┐ ┌───────────────────┐ │ Crawly.Manager ├────────> Crawly.ManagerSup │ └────────────────┘ └─────────┬─────────┘

     │                          |
     │                          |
     ┌──────────────────────────┤
     │                          │
     │                          │

┌────────▼───────┐ ┌─────────▼───────┐ │ Worker1 │ │ Worker2 │ └────────┬───────┘ └────────┬────────┘

     │                         │
     │                         │
     │                         │
     │                         │

┌────────▼─────────┐ ┌──────────▼───────────┐ │Crawly.DataStorage│ │Crawly.RequestsStorage│ └──────────────────┘ └──────────────────────┘

Link to this section Summary

Functions

add_workers(spider_name, num_of_workers)

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

start_link(list)

Link to this section Functions

add_workers(spider_name, num_of_workers)

Specs

add_workers(module(), non_neg_integer()) :: :ok | {:error, :spider_non_exist}

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.