Crawler.Fetcher.Modifier (Crawler v1.1.2) View Source

Modifies request options and headers before dispatch.

Link to this section Summary

Functions

Allows modifying headers prior to making a crawl request.

Allows passing opts to httpPoison prior to making the crawl request

Link to this section Functions

Allows modifying headers prior to making a crawl request.

Example implementation

def headers(opts) do
  if opts[:url] == "http://modifier" do
    [{"Referer", "http://fetcher"}]
  end
  []
end

Allows passing opts to httpPoison prior to making the crawl request

Example implementation

def opts(opts) do
  if opts[:url] == "http://modifier" do
    # add a new pool to hackney
    [hackney: [pool: :modifier]]
  end
  []
end