Essence.Token (essence v0.3.0)
This module contains helpful functions around Tokens
Link to this section Summary
Functions
Calculates the average word length over the given list of tokens.
Count the number of syllabels in a token.
Determines whether a given token is a polysyllabic, containing more than n syllables, or not.
Determines whether a given token is punctuation or not.
Returns true if the token is punctuation, false otherwise.
Determines whether a given token is a word or not. Returns true if the
token is a word, false otherwise.
Calculates the length of a token as the number of graphemes in that token.
Link to this section Functions
avg_word_length(tokens)
Calculates the average word length over the given list of tokens.
count_syllabels(token)
Count the number of syllabels in a token.
is_polysyllabic?(token, n \\ 1)
Determines whether a given token is a polysyllabic, containing more than n syllables, or not.
is_punctuation?(token)
Determines whether a given token is punctuation or not.
Returns true if the token is punctuation, false otherwise.
is_word?(token)
Determines whether a given token is a word or not. Returns true if the
token is a word, false otherwise.
token_length(token)
Calculates the length of a token as the number of graphemes in that token.