Crawler v1.1.0 Crawler.Fetcher.Modifier

Modifies request options and headers before dispatch.

Link to this section Summary

Allows modifing headers prior to making a crawl request

Allows passing opts to httpPoison prior to making the crawl request

Allows modifing headers prior to making a crawl request.

def headers(opts) do

if opts[:url] == "http://modifier" do
  [{"Referer", "http://fetcher"}]
end
[]

end

Allows passing opts to httpPoison prior to making the crawl request

def opts(opts) do

if opts[:url] == "http://modifier" do
  # add a new pool to hackney
  [hackney: [pool: :modifier]]
end
[]

end