Gollum.Fetcher (gollum v0.5.0)

In charge of fetching the actual robots.txt files.

Summary

Fetches the robots.txt file from the specified host. Simply performs a GET request to the domain via HTTPoison.

@spec fetch(
  binary(),
  keyword()
) :: {:ok, binary()} | {:error, term()}

Fetches the robots.txt file from the specified host. Simply performs a GET request to the domain via HTTPoison.