Crawly.DataStorage (Crawly v0.17.2) View Source

Data Storage, is a module responsible for storing crawled items. On the high level it's possible to represent the architecture of items storage this way:

           ┌──────────────────┐
           │                  │             ┌------------------┐
           │   DataStorage    <─────────────┤ From crawlers1,2 │
           │                  │             └------------------┘
           └─────────┬────────┘
                     │
                     │
                     │
                     │
        ┌────────────▼─────────────────┐
        │                              │
        │                              │
        │                              │

┌───────────▼──────────┐ ┌───────────▼──────────┐ │ DataStorageWorker1 │ │ DataStorageWorker2 │ │ (Crawler1) │ │ (Crawler2) │ └──────────────────────┘ └──────────────────────┘

Link to this section Summary

Functions

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

init(args)

Callback implementation for GenServer.init/1.

inspect(spider, field)

start_link(list)

start_worker(spider_name, crawl_id)

stats(spider)

store(spider, item)

Link to this section Functions