gollum

v0.3.3

gollum v0.3.3 Gollum.Host

Represents one host's robots.txt files.

Link to this section Summary

Types

t()

Functions

crawlable?(host, user_agent, path)

Returns whether a specified path is crawlable by the specified user agent, based on the rules defined in the specified host struct

new(host, rules)

Creates a new Gollum.Host struct, passing in the host and rules. The rules usually are the output of the parser

Link to this section Types

Link to this type

t() 
t() :: %Gollum.Host{host: binary(), rules: map()}

Link to this section Functions

Link to this function

crawlable?(host, user_agent, path) 
crawlable?(Gollum.Host.t(), binary(), binary()) ::
  :crawlable | :uncrawlable | :undefined

Returns whether a specified path is crawlable by the specified user agent, based on the rules defined in the specified host struct.

Checks are done based on the specification defined by Google, which can be found here.

Examples 

iex> alias Gollum.Host
iex> rules = %{
...>   "hello" => %{
...>     allowed: ["/p"],
...>     disallowed: ["/"],
...>   },
...>   "otherhello" => %{
...>     allowed: ["/$"],
...>     disallowed: ["/"],
...>   },
...>   "*" => %{
...>     allowed: ["/page"],
...>     disallowed: ["/*.htm"],
...>   },
...> }
iex> host = Host.new("hello.net", rules)
iex> Host.crawlable?(host, "Hello", "/page")
:crawlable
iex> Host.crawlable?(host, "OtherHello", "/page.htm")
:uncrawlable
iex> Host.crawlable?(host, "NotHello", "/page.htm")
:undefined
Link to this function

new(host, rules) 
new(binary(), map()) :: Gollum.Host.t()

Creates a new Gollum.Host struct, passing in the host and rules. The rules usually are the output of the parser.

Examples 

iex> alias Gollum.Host
iex> rules = %{"Hello" => %{allowed: [], disallowed: []}}
iex> Host.new("hello.net", rules)
%Gollum.Host{host: "hello.net", rules: %{"Hello" => %{allowed: [], disallowed: []}}}