Crawly.Middlewares.RequestOptions (Crawly v0.17.2) View Source

Request settings middleware

Allows to specify HTTP request settings like follow_redirect, or request timeout.

If using HTTPoisonFetcher (the default), please refer to the HTTPoison Request documentation for full list of :options.

Example Usage

Example Declaration

middlewares: [
  {Crawly.Middlewares.RequestOptions, [timeout: 30_000, recv_timeout: 15000]}
]

Declaring proxy settings

middlewares: [
 {Crawly.Middlewares.RequestOptions, [proxy: {"https://my_host.com", 3000}, proxy_auth: {"my_user", "my_password}]}
]

Link to this section Summary

Link to this section Functions

Link to this function

run(request, state, options \\ [])

View Source

Callback implementation for Crawly.Pipeline.run/3.