Crawly.DataStorage (Crawly v0.13.0) View Source

Data Storage, is a module responsible for storing crawled items. On the high level it's possible to represent the architecture of items storage this way:

           
                                          ------------------
              DataStorage    < From crawlers1,2 
                                          ------------------
           
                     
                     
                     
                     
        
                                      
                                      
                                      

┌───────────▼──────────┐ ┌───────────▼──────────┐ │ DataStorageWorker1 │ │ DataStorageWorker2 │ │ (Crawler1) │ │ (Crawler2) │ └──────────────────────┘ └──────────────────────┘

Link to this section Summary

Functions

Returns a specification to start this module under a supervisor.

Callback implementation for GenServer.init/1.

Link to this section Functions

Returns a specification to start this module under a supervisor.

See Supervisor.

Callback implementation for GenServer.init/1.

Link to this function

start_worker(spider_name, crawl_id)

View Source

Specs

store(atom(), map()) :: :ok