Crawly.Pipelines.DuplicatesFilter (Crawly v0.17.0) View Source

Filters out duplicated items based on the provided item_id.

Stores identifier values in state under the :duplicates_filter key.


If item unique identifier is not provided, this pipeline does nothing.

  • :item_id, required: Designates a field to be used to check for duplicates. Falls back to global config :item_id.

Example Usage

  iex> item = %{my: "item"}
  iex> {_unchanged, new_state} =, %{}, item_id: :my)

  # Rerunning the item through the pipeline will drop the item
  iex>, %{}, item_id: :id)
  {false, %{
    duplicates_filter: %{"item" => true}