Crawly.DataStorage.Worker (Crawly v0.17.2) View Source
A worker process which stores items for individual spiders. All items are pre-processed by item_pipelines.
All pipelines are using the state of this process for their internal needs (persistancy).
The DataStorage.Worker will not write anything to a filesystem. Instead it would expect that pipelines are going to do that work.
Link to this section Summary
Functions
Returns a specification to start this module under a supervisor.
Callback implementation for GenServer.init/1.
Inspect the inner state of the given data worker
Link to this section Functions
Returns a specification to start this module under a supervisor.
See Supervisor.
Callback implementation for GenServer.init/1.
Specs
Inspect the inner state of the given data worker
Specs
stats(pid()) :: {:stored_items, non_neg_integer()}