Crawly.DataStorage (Crawly v0.17.2) View Source

Data Storage, is a module responsible for storing crawled items. On the high level it's possible to represent the architecture of items storage this way:

           
                                          ------------------
              DataStorage    < From crawlers1,2 
                                          ------------------
           
                     
                     
                     
                     
        
                                      
                                      
                                      

┌───────────▼──────────┐ ┌───────────▼──────────┐ │ DataStorageWorker1 │ │ DataStorageWorker2 │ │ (Crawler1) │ │ (Crawler2) │ └──────────────────────┘ └──────────────────────┘

Link to this section Summary

Link to this section Functions

Returns a specification to start this module under a supervisor.

See Supervisor.

Callback implementation for GenServer.init/1.

Specs

inspect(atom(), term()) :: {:error, :data_storage_worker_not_running} | term()
Link to this function

start_worker(spider_name, crawl_id)

View Source

Specs

store(atom(), map()) :: :ok