tokenizer

The tokenizer converts the plural forms syntax into a list of tokens for later parsing.

Types

Token

pub type Token {
  N
  NPlurals
  Plural
  Equals
  NotEquals
  GreaterThanOrEquals
  LowerThanOrEquals
  GreaterThan
  LowerThan
  Assignment
  Ternary
  TernaryElse
  Remainder
  Or
  And
  Semicolon
  LParen
  RParen
  End
  Int(value: Int)
}

Constructors

```
N
```
```
NPlurals
```
```
Plural
```
```
Equals
```
```
NotEquals
```
```
GreaterThanOrEquals
```
```
LowerThanOrEquals
```
```
GreaterThan
```
```
LowerThan
```
```
Assignment
```
```
Ternary
```
```
TernaryElse
```
```
Remainder
```
```
Or
```
```
And
```
```
Semicolon
```
```
LParen
```
```
RParen
```
```
End
```
```
Int(value: Int)
```

Values

tokenize

</>

pub fn tokenize(
  str: String,
) -> Result(
  List(Token(kielet/plurals/tokenizer.Token)),
  SyntaxError,
)

Tokenize the given plural forms syntax.

Whitespace is ignored and backslashes at the end of lines are removed.