Crawler v1.1.2

Crawler.Fetcher.Modifier (Crawler v1.1.2) View Source

Modifies request options and headers before dispatch.

Link to this section Summary

Functions

headers(opts)

Allows modifying headers prior to making a crawl request.

opts(opts)

Allows passing opts to httpPoison prior to making the crawl request

Link to this section Functions

Link to this function

headers(opts)

View Source

Allows modifying headers prior to making a crawl request.

Example implementation 

def headers(opts) do
  if opts[:url] == "http://modifier" do
    [{"Referer", "http://fetcher"}]
  end
  []
end
Link to this function

opts(opts)

View Source

Allows passing opts to httpPoison prior to making the crawl request

Example implementation 

def opts(opts) do
  if opts[:url] == "http://modifier" do
    # add a new pool to hackney
    [hackney: [pool: :modifier]]
  end
  []
end