Crawly.DataStorage (Crawly v0.17.0) View Source

Data Storage, is a module responsible for storing crawled items. On the high level it's possible to represent the architecture of items storage this way:

              DataStorage    < From crawlers1,2 

┌───────────▼──────────┐ ┌───────────▼──────────┐ │ DataStorageWorker1 │ │ DataStorageWorker2 │ │ (Crawler1) │ │ (Crawler2) │ └──────────────────────┘ └──────────────────────┘

Link to this section Summary

Link to this section Functions

Returns a specification to start this module under a supervisor.

See Supervisor.

Callback implementation for GenServer.init/1.


inspect(atom(), term()) :: {:error, :data_storage_worker_not_running} | term()
Link to this function

start_worker(spider_name, crawl_id)

View Source


store(atom(), map()) :: :ok