Crawly.Middlewares.UniqueRequest (Crawly v0.17.2) View Source

Avoid scheduling multiple requests for the same page. Allow to set a hashing algorithm via options to reduce the memory footprint. Be aware of reduced collision resistance, depending on the chosen algorithm.

Example Declarations

middlewares: [
  Crawly.Middlewares.UniqueRequest
]
middlewares: [
  {Crawly.Middlewares.UniqueRequest, hash: :sha}
]

See the Erlang documentation for crypto for available algorithms.

Link to this section Summary

Link to this section Functions

Link to this function

run(request, state, opts \\ [])

View Source