gollum

v0.3.3

gollum v0.3.3 Gollum.Fetcher

In charge of fetching the actual robots.txt files.

Link to this section Summary

Functions

fetch(domain, opts)

Fetches the robots.txt file from the specified host. Simply performs a GET request to the domain via HTTPoison

Link to this section Functions

Link to this function

fetch(domain, opts) 
fetch(binary(), keyword()) :: {:ok, binary()} | {:error, term()}

Fetches the robots.txt file from the specified host. Simply performs a GET request to the domain via HTTPoison.