View Source SpiderMan.Component.Spider (spider_man v0.6.3)
Analyze web pages.
Life cycle of request:
- insert responses to ets of spider component.
- component's producer get pass out to processes.
- processes handle message.
- handle by pre pipelines.
- call
SpiderModule.handle_response/2
.
- pass out message.
- if success, pass out requests to Downloader component's ets.
- if success, pass out items to ItemProcessor component's ets.
- if failed, maybe try again.