Crawly.Middlewares.UserAgent (Crawly v0.17.2) View Source

Set/Rotate user agents for crawling. The user agents are read from :crawly, :user_agents sessions.

The default value for the user agent is: Crawly Bot 1.0

Rotation is determined through Enum.random/1.

Options

:user_agents, optional. A list of user agent strings to rotate. Defaults to "Crawly Bot 1.0".

Example Declaration

middlewares: [
  {UserAgent, user_agents: ["My Custom Bot"] }
]

Link to this section Summary

Functions

run(request, state, opts \\ [])

Link to this section Functions

run(request, state, opts \\ [])