Crawly.Spider behaviour (Crawly v0.16.0) View Source
A behavior module for implementing a Crawly Spider
A Spider is a module which is responsible for defining:
init/0function, which must return a keyword list with start_urls/start_requests list
init/1same as init, but also takes a list of options sent from Engine
base_url/0function responsible for filtering out requests not related to a given website
override_settings/0function that is called each time a setting is referenced internally. Allows overriding of Crawly configuration at the spider-level.
parse_item/1function which is responsible for parsing the downloaded request and converting it into items which can be stored and new requests which can be scheduled
custom_settings/0an optional callback which can be used in order to provide custom spider specific settings. Should define a list with custom settings and their values. These values will take precedence over the global settings defined in the config.
Link to this section Summary
Link to this section Callbacks
base_url() :: binary()
override_settings() :: Crawly.Settings.t()