Floki v0.23.0 Floki View Source

Floki is a simple HTML parser that enables search for nodes using CSS selectors.

Example

Assuming that you have the following HTML:

<!doctype html>
<html>
<body>
  <section id="content">
    <p class="headline">Floki</p>
    <a href="http://github.com/philss/floki">Github page</a>
    <span data-model="user">philss</span>
  </section>
</body>
</html>

Examples of queries that you can perform:

  • Floki.find(html, "#content")
  • Floki.find(html, ".headline")
  • Floki.find(html, "a")
  • Floki.find(html, "[data-model=user]")
  • Floki.find(html, "#content a")
  • Floki.find(html, ".headline, a")

Each HTML node is represented by a tuple like:

{tag_name, attributes, children_nodes}

Example of node:

{"p", [{"class", "headline"}], ["Floki"]}

So even if the only child node is the element text, it is represented inside a list.

You can write a simple HTML crawler (with support of HTTPoison) with a few lines of code:

html
|> Floki.find(".pages a")
|> Floki.attribute("href")
|> Enum.map(fn(url) -> HTTPoison.get!(url) end)

It is simple as that!

Link to this section Summary

Functions

Changes the attribute values of the elements matched by selector with the function mutation and returns the whole element tree

Returns a list with attribute values from elements.

Returns a list with attribute values for a given selector.

Returns the nodes from a HTML tree that don't match the filter selector.

Find elements inside a HTML tree or string.

It receives a HTML tree structure as tuple and maps through all nodes with a given function that receives a tuple with {name, attributes}.

Parses a HTML string.

Converts HTML tree to raw HTML. Note that the resultant HTML may be different from the original one. Spaces after tags and doctypes are ignored.

Returns the text nodes from a HTML tree. By default, it will perform a deep search through the HTML tree. You can disable deep search with the option deep assigned to false. You can include content of script tags with the option js assigned to true. You can specify a separator between nodes content.

Traverses a HTML tree structure and returns a new tree structure that is the result of executing a function on all nodes. The function receives a tuple with {name, attributes, children}, and should either return a similar tuple or nil to delete the current node.

Link to this section Types

Link to this type

html_tree()

View Source
html_tree() :: tuple() | list()

Link to this section Functions

Link to this function

attr(html_elem_tuple, selector, attribute_name, mutation)

View Source
attr(binary() | html_tree(), binary(), binary(), (binary() -> binary())) ::
  html_tree()

Changes the attribute values of the elements matched by selector with the function mutation and returns the whole element tree

Examples

iex> Floki.attr("<div id='a'></div>", "#a", "id", fn(id) -> String.replace(id, "a", "b") end)
[{"div", [{"id", "b"}], []}]

iex> Floki.attr("<div class='class_name'></div>", "div", "id", fn _ -> "b" end)
[{"div", [{"id", "b"}, {"class", "class_name"}], []}]
Link to this function

attribute(html_tree, attribute_name)

View Source
attribute(binary() | html_tree(), binary()) :: list()

Returns a list with attribute values from elements.

Examples

iex> Floki.attribute("<a href=https://google.com>Google</a>", "href")
["https://google.com"]

iex> Floki.attribute([{"a", [{"href", "https://google.com"}], ["Google"]}], "href")
["https://google.com"]
Link to this function

attribute(html, selector, attribute_name)

View Source
attribute(binary() | html_tree(), binary(), binary()) :: list()

Returns a list with attribute values for a given selector.

Examples

iex> Floki.attribute("<a href='https://google.com'>Google</a>", "a", "href")
["https://google.com"]

iex> Floki.attribute([{"a", [{"href", "https://google.com"}], ["Google"]}], "a", "href")
["https://google.com"]
Link to this function

filter_out(html_tree, selector)

View Source
filter_out(binary() | html_tree(), Floki.FilterOut.selector()) :: list()

Returns the nodes from a HTML tree that don't match the filter selector.

Examples

iex> Floki.filter_out("<div><script>hello</script> world</div>", "script")
{"div", [], [" world"]}

iex> Floki.filter_out([{"body", [], [{"script", [], []},{"div", [], []}]}], "script")
[{"body", [], [{"div", [], []}]}]

iex> Floki.filter_out("<div><!-- comment --> text</div>", :comment)
{"div", [], [" text"]}
Link to this function

find(html, selector)

View Source
find(binary() | html_tree(), binary()) :: html_tree()

Find elements inside a HTML tree or string.

Examples

iex> Floki.find("<p><span class=hint>hello</span></p>", ".hint")
[{"span", [{"class", "hint"}], ["hello"]}]

iex> Floki.find("<body><div id=important><div>Content</div></div></body>", "#important")
[{"div", [{"id", "important"}], [{"div", [], ["Content"]}]}]

iex> Floki.find("<p><a href='https://google.com'>Google</a></p>", "a")
[{"a", [{"href", "https://google.com"}], ["Google"]}]

iex> Floki.find([{ "div", [], [{"a", [{"href", "https://google.com"}], ["Google"]}]}], "div a")
[{"a", [{"href", "https://google.com"}], ["Google"]}]
Link to this function

map(html_tree_list, fun)

View Source

It receives a HTML tree structure as tuple and maps through all nodes with a given function that receives a tuple with {name, attributes}.

It returns that structure transformed by the function.

Examples

iex> html = {"div", [{"class", "foo"}], ["text"]}
iex> Floki.map(html, fn({name, attrs}) -> {name, [{"data-name", "bar"} | attrs]} end)
{"div", [{"data-name", "bar"}, {"class", "foo"}], ["text"]}

Parses a HTML string.

Examples

iex> Floki.parse("<div class=js-action>hello world</div>")
{"div", [{"class", "js-action"}], ["hello world"]}

iex> Floki.parse("<div>first</div><div>second</div>")
[{"div", [], ["first"]}, {"div", [], ["second"]}]
Link to this function

raw_html(html_tree, options \\ [])

View Source
raw_html(html_tree() | binary(), keyword()) :: binary()

Converts HTML tree to raw HTML. Note that the resultant HTML may be different from the original one. Spaces after tags and doctypes are ignored.

Floki.raw_html/2 accepts a keyword list of options. Currently, the only supported option is :encode, which can be set to true or false.

You can also control the encoding behaviour at the application level via config :floki, :encode_raw_html, true | false

Examples

iex> Floki.parse(~s(<div class="wrapper">my content</div>)) |> Floki.raw_html
~s(<div class="wrapper">my content</div>)

iex> Floki.parse(~s(<div class="wrapper">10 > 5</div>)) |> Floki.raw_html(encode: true)
~s(<div class="wrapper">10 &gt; 5</div>)

iex> Floki.parse(~s(<div class="wrapper">10 > 5</div>)) |> Floki.raw_html(encode: false)
~s(<div class="wrapper">10 > 5</div>)
Link to this function

text(html, opts \\ [deep: true, js: false, style: true, sep: ""])

View Source

Returns the text nodes from a HTML tree. By default, it will perform a deep search through the HTML tree. You can disable deep search with the option deep assigned to false. You can include content of script tags with the option js assigned to true. You can specify a separator between nodes content.

Examples

iex> Floki.text("<div><span>hello</span> world</div>")
"hello world"

iex> Floki.text("<div><span>hello</span> world</div>", deep: false)
" world"

iex> Floki.text("<div><script>hello</script> world</div>")
" world"

iex> Floki.text("<div><script>hello</script> world</div>", js: true)
"hello world"

iex> Floki.text("<ul><li>hello</li><li>world</li></ul>", sep: " ")
"hello world"

iex> Floki.text([{"div", [], ["hello world"]}])
"hello world"

iex> Floki.text([{"p", [], ["1"]},{"p", [], ["2"]}])
"12"

iex> Floki.text("<div><style>hello</style> world</div>")
"hello world"

iex> Floki.text("<div><style>hello</style> world</div>", style: false)
" world"
Link to this function

traverse_and_update(html_tree, fun)

View Source

Traverses a HTML tree structure and returns a new tree structure that is the result of executing a function on all nodes. The function receives a tuple with {name, attributes, children}, and should either return a similar tuple or nil to delete the current node.

Examples

iex> html = {"div", [], ["hello"]}
iex> Floki.traverse_and_update(html, fn {"div", attrs, children} -> {"p", attrs, children} end)
{"p", [], ["hello"]}

iex> html = {"div", [], [{"span", [], ["hello"]}]}
iex> Floki.traverse_and_update(html, fn {"span", _attrs, _children} -> nil; tag -> tag end)
{"div", [], []}