Avoid scheduling multiple requests for the same page. Allow to set a hashing algorithm via options to reduce the memory footprint. Be aware of reduced collision resistance, depending on the chosen algorithm.

Example Declarations

middlewares: [
  Crawly.Middlewares.UniqueRequest
]

middlewares: [
  {Crawly.Middlewares.UniqueRequest, hash: :sha}
]

See the Erlang documentation for crypto for available algorithms.