Crawly.Middlewares.DomainFilter (Crawly v0.17.0) View Source

Filters out requests which are going outside of the crawled domain.

The domain that is used to compare against the request url is obtained from the spider's c:Crawly.Spider.base_url callback.

Does not accept any options. Tuple-based configuration options will be ignored.

Example Declaration

middlewares: [
  Crawly.Middlewares.DomainFilter
]

Link to this section Summary

Link to this section Functions

Link to this function

run(request, state, opts \\ [])

View Source

Callback implementation for Crawly.Pipeline.run/3.