Crawly.Manager (Crawly v0.17.0) View Source

Crawler manager module

This module is responsible for spawning all processes related to a given Crawler.

The manager spawns the following processes tree. ┌────────────────┐ ┌───────────────────┐ │ Crawly.Manager ├────────> Crawly.ManagerSup │ └────────────────┘ └─────────┬─────────┘

                               |
                               |
     
                               
                               

┌────────▼───────┐ ┌─────────▼───────┐ │ Worker1 │ │ Worker2 │ └────────┬───────┘ └────────┬────────┘

                              
                              
                              
                              

┌────────▼─────────┐ ┌──────────▼───────────┐ │Crawly.DataStorage│ │Crawly.RequestsStorage│ └──────────────────┘ └──────────────────────┘

Link to this section Summary

Functions

Returns a specification to start this module under a supervisor.

Link to this section Functions

Link to this function

add_workers(spider_name, num_of_workers)

View Source

Specs

add_workers(module(), non_neg_integer()) :: :ok | {:error, :spider_non_exist}

Returns a specification to start this module under a supervisor.

See Supervisor.