ExNlp.Filter.Stopwords (ex_nlp v0.1.0)

Stopword filtering for tokens.

Removes stop words from token streams using ExNlp.Stopwords.

Summary

Types

language()

Supported language atoms

token()

A token struct

Functions

filter(tokens, lang \\ :english)

Removes stop words from the token stream.

Types

language()

@type language() :: atom()

Supported language atoms

token()

@type token() :: ExNlp.Token.t()

A token struct

Functions

filter(tokens, lang \\ :english)

@spec filter([token()], language()) :: [token()]

Removes stop words from the token stream.

Supports multiple languages. Defaults to English if language not specified.

Examples

iex> tokens = [%ExNlp.Token{text: "the"}, %ExNlp.Token{text: "quick"}, %ExNlp.Token{text: "brown"}]
iex> ExNlp.Filter.Stopwords.filter(tokens, :english)
[%ExNlp.Token{text: "quick"}, %ExNlp.Token{text: "brown"}]