View Source SpiderMan.Component.Spider (spider_man v0.6.3)

Analyze web pages.

Life cycle of request:

  1. insert responses to ets of spider component.
  2. component's producer get pass out to processes.
  3. processes handle message.
    1. handle by pre pipelines.
    2. call SpiderModule.handle_response/2.
  4. pass out message.
    1. if success, pass out requests to Downloader component's ets.
    2. if success, pass out items to ItemProcessor component's ets.
    3. if failed, maybe try again.

Summary

Functions

start_link(options)