Crawly.Pipelines.Validate (Crawly v0.17.2) View Source

Ensure that scraped item contains a set of required fields.

Options

If the fields to check are not provided, the pipeline does nothing.

• :fields, required: The list of required fields. Fallsback to global config :item.

Example Declaration

pipelines: [
  {Crawly.Pipelines.Validate, fields: [:id, :url, :date]}
]

Example Usage

# Drops the scraped item that does not have the required fields
iex> Validate.run(%{my: "field"}, %{}, fields: [:id])
{false, %{}}