View Source Crawler.Fetcher.Modifier (Crawler v1.5.0)

Modifies request options and headers before dispatch.

Summary

headers(opts)

Allows modifying headers prior to making a crawl request.

opts(opts)

Allows passing opts to httpPoison prior to making the crawl request

Allows modifying headers prior to making a crawl request.

def headers(opts) do
  if opts[:url] == "http://modifier" do
    [{"Referer", "http://fetcher"}]
  end
  []
end

Allows passing opts to httpPoison prior to making the crawl request

def opts(opts) do
  if opts[:url] == "http://modifier" do
    # add a new pool to hackney
    [hackney: [pool: :modifier]]
  end
  []
end