ExNlp.Filter.Length (ex_nlp v0.1.0)

Length-based token filtering.

Filters tokens based on minimum and maximum length constraints.

Summary

Types

token()

A token struct

Functions

maximum(tokens, max_len \\ 50)

Removes tokens longer than the specified maximum length.

minimum(tokens, min_len \\ 2)

Removes tokens shorter than the specified minimum length.

Types

token()

@type token() :: ExNlp.Token.t()

A token struct

Functions

maximum(tokens, max_len \\ 50)

@spec maximum([token()], non_neg_integer()) :: [token()]

Removes tokens longer than the specified maximum length.

Examples

iex> tokens = [%ExNlp.Token{text: "short"}, %ExNlp.Token{text: "verylongword"}]
iex> ExNlp.Filter.Length.maximum(tokens, 5)
[%ExNlp.Token{text: "short"}]

minimum(tokens, min_len \\ 2)

@spec minimum([token()], non_neg_integer()) :: [token()]

Removes tokens shorter than the specified minimum length.

Examples

iex> tokens = [%ExNlp.Token{text: "a"}, %ExNlp.Token{text: "an"}, %ExNlp.Token{text: "the"}]
iex> ExNlp.Filter.Length.minimum(tokens, 2)
[%ExNlp.Token{text: "an"}, %ExNlp.Token{text: "the"}]