Crawly.Pipelines.Validate (Crawly v0.17.0) View Source

Ensure that scraped item contains a set of required fields.


If the fields to check are not provided, the pipeline does nothing.

  • :fields, required: The list of required fields. Fallsback to global config :item.

Example Declaration

pipelines: [
  {Crawly.Pipelines.Validate, fields: [:id, :url, :date]}

Example Usage

# Drops the scraped item that does not have the required fields
iex>{my: "field"}, %{}, fields: [:id])
{false, %{}}