View Source SpiderMan.Component.ItemProcessor (spider_man v0.6.3)

Store items.

Life cycle of request:

  1. insert requests to ets of downloader.
  2. downloader's producer get pass out to processes.
  3. processes handle message.
    1. handle by pre pipelines.
    2. call Requester.request/3.
    3. handle by post pipelines.
  4. pass out message.
    1. if success, pass out to Batchers.
    2. if failed, maybe try again.
  5. batcher get enough message and call handle_batch/4.
    1. call Storage.store/3 to save items.
    2. if success, done.
    3. if failed, maybe try again.

Summary

Functions

start_link(options)