gollum v0.3.3 Gollum.Parser

Parses a robots.txt file.

Link to this section Summary

Functions

Parse the file, passed in as a simple binary

Link to this section Functions

parse(string)

parse(binary()) :: map()

Parse the file, passed in as a simple binary.

It follows the spec defined by Google as closely as possible.

Examples

iex> alias Gollum.Parser
iex> Parser.parse("User-agent: Hello\nAllow: /hello\nDisallow: /hey")
%{"hello" => %{allowed: ["/hello"], disallowed: ["/hey"]}}