Crawly

v0.9.0

Crawly

Crawly v0.9.0 Crawly.Settings View Source

Define Crawly setting types

Link to this section Summary

Types

middleware()
numeric_setting()
pipeline()
retry()
t()

Link to this section Types

Link to this type

middleware() View Source 
middleware() ::
  Crawly.Middlewares.DomainFilter
  | Crawly.Middlewares.UniqueRequest
  | Crawly.Middlewares.RobotsTxt
  | Crawly.Middlewares.AutoCookiesManager
  | {Crawly.Middlewares.UserAgent, [{:user_agents, [binary()]}]}

Link to this type

numeric_setting() View Source 
numeric_setting() :: pos_integer() | :disabled

Link to this type

pipeline() View Source 
pipeline() ::
  Crawly.Pipelines.JSONEncoder
  | {Crawly.Pipelines.DuplicatesFilter, [{:item_id, atom()}]}
  | {Crawly.Pipelines.Validate, [{:fields, [atom()]}]}
  | {Crawly.Pipelines.CSVEncoder, [{:fields, [atom()]}]}
  | {Crawly.Pipelines.WriteToFile, [folder: binary(), extension: binary()]}

Link to this type

retry() View Source 
retry() :: [
  retry_codes: [pos_integer()],
  max_retries: pos_integer(),
  ignored_middlewares: [module()]
]

Link to this type

t() View Source 
t() :: [
  closespider_itemcount: numeric_setting(),
  closespider_timeout: pos_integer(),
  concurrent_requests_per_domain: pos_integer(),
  fetcher: Crawly.Fetchers.Fetcher.t(),
  retry: retry(),
  middlewares: [middleware()],
  pipelines: [pipeline()],
  on_spider_closed_callback: function()
]