Text.Vocabulary behaviour (Text v0.2.0) View Source
A vocabulary is the encoded form of a training text that is used to support language matching.
A vocabulary is mapping of an n-gram to its rank and probability.
Link to this section Summary
Functions
Calculate the n-grams for a given text
Returns the ngrams for a given text and range representing a range of n-grams
Get the vocabulary entry for a given language and vocabulary
Loads the given vocabulary.
Returns the top n by rank for a list of entries for a given languages vocabulary
Rerturns a list of the top n vocabulary entries by rank for a given language and vocabulary.
Link to this section Types
Specs
t() :: module()
Link to this section Callbacks
Link to this section Functions
Calculate the n-grams for a given text
A range of n-grams is calculated from
range and the top n ranked
n-grams from the text are returned
Returns the ngrams for a given text and range representing a range of n-grams
Get the vocabulary entry for a given language and vocabulary
Loads the given vocabulary.
Vocabularies are placed in
:persistent_store since this
reduces memory copies and has efficient
multi-process access.
Returns the top n by rank for a list of entries for a given languages vocabulary
Rerturns a list of the top n vocabulary entries by rank for a given language and vocabulary.
This function is primarily intended for debugging support.