glexer

Types

Lexer

opaque </>

pub opaque type Lexer

Position

</>

pub type Position {
  Position(byte_offset: Int)
}

Constructors

```
Position(byte_offset: Int)
```
Arguments

byte_offset

Despite being named ‘byte_offset’, this isn’t actually a ‘byte_offset’ on JavaScript. Strings on JavaScript are utf-16, so this is actually more like a ‘codeunit’ offset.

Values

discard_comments

</>

pub fn discard_comments(lexer: Lexer) -> Lexer

discard_whitespace

</>

pub fn discard_whitespace(lexer: Lexer) -> Lexer

lex

</>

pub fn lex(lexer: Lexer) -> List(#(token.Token, Position))

new

</>

pub fn new(source: String) -> Lexer

to_source

</>

pub fn to_source(
  tokens: List(#(token.Token, Position)),
) -> String

Turn a sequence of tokens back to their Gleam source code representation.

unescape_string

</>

pub fn unescape_string(string: String) -> Result(String, Nil)

Convert the value of a string token to the string it represents.

This function can fail if the original string contains invalid escape sequences.

unescape_string("\\\"X\\\" marks the spot")
// --> Ok("\"X\" marks the spot")

unescape_string("\\u{1F600}")
// --> Ok("😀")

unescape_string("\\x")
// --> Error(Nil)

Constructors

Arguments