Crawly.Middlewares.DomainFilter (Crawly v0.17.2) View Source
Filters out requests which are going outside of the crawled domain.
The domain that is used to compare against the request url is obtained from the spider's c:Crawly.Spider.base_url
callback.
Does not accept any options. Tuple-based configuration options will be ignored.
Example Declaration
middlewares: [
Crawly.Middlewares.DomainFilter
]
Link to this section Summary
Functions
Callback implementation for Crawly.Pipeline.run/3
.
Link to this section Functions
Callback implementation for Crawly.Pipeline.run/3
.