View Source Haystack.Transformer behaviour (Haystack v0.1.0)
A module for transforming tokens.
This module includes a transform/1 callback behaviour for implementing stages of a transformation pipeline. The default pipeline transformers includes stages for stemming and removing stop words.
Link to this section Summary
Callbacks
Apply a transformation on a list of tokens.
Functions
Return the list of default transformers.
Apply a pipeline of transformers to a list of tokens.
Link to this section Callbacks
@callback transform([Haystack.Tokenizer.Token.t()]) :: [Haystack.Tokenizer.Token.t()]
Apply a transformation on a list of tokens.
Link to this section Functions
@spec default() :: [module()]
Return the list of default transformers.
examples
Examples
iex> Transformer.default()
[Transformer.Stemmer, Transformer.StopWords]@spec pipeline([Haystack.Tokenizer.Token.t()], [module()]) :: [ Haystack.Tokenizer.Token.t() ]
Apply a pipeline of transformers to a list of tokens.
examples
Examples
iex> tokens = Tokenizer.tokenize("Needle in a Haystack")
iex> tokens = Transformer.pipeline(tokens, Transformer.default())
iex> Enum.map(tokens, & &1.v)
~w{needl haystack}