tear

Types

Base

pub type Base {
  Binary
  Octal
  Decimal
  Hexadecimal
}

Constructors

```
Binary
```
```
Octal
```
```
Decimal
```
```
Hexadecimal
```

HighlightToken

</>

HighlightTokens stringify, classify and flatten the lexer tokens into simpler groups.

pub type HighlightToken {
  HighlightError(str: String)
  HighlightKeyword(str: String)
  HighlightBuiltin(str: String)
  HighlightAttribute(str: String)
  HighlightAlias(str: String)
  HighlightFunction(str: String)
  HighlightIdentifier(str: String)
  HighlightNumber(str: String)
  HighlightString(str: String)
  HighlightLiteral(str: String)
  HighlightAtom(str: String)
  HighlightOperator(str: String)
  HighlightPunctuation(str: String)
  HighlightComment(str: String)
  HighlightWhitespace(str: String)
}

Constructors

```
HighlightError(str: String)
```
```
HighlightKeyword(str: String)
```
A true keyword, like do, after, in or when
```
HighlightBuiltin(str: String)
```
A builtin, like defp, import, or __ENV__
```
HighlightAttribute(str: String)
```
An attribute, like @spec
```
HighlightAlias(str: String)
```
An alias, like Elixir
```
HighlightFunction(str: String)
```
An identifier in a position that is likely a call.
```
HighlightIdentifier(str: String)
```
An identifier that is not otherwise recognised as significant.
```
HighlightNumber(str: String)
```
An integer or floating-point number
```
HighlightString(str: String)
```
A string, sigil or heredoc.
```
HighlightLiteral(str: String)
```
nil, true, and false
```
HighlightAtom(str: String)
```
An atom or keyword identifier.
```
HighlightOperator(str: String)
```
An operator, including arrows.
```
HighlightPunctuation(str: String)
```
Punctuation marks, like commas or parenthesis
```
HighlightComment(str: String)
```
A comment or @doc attribute, or a discarded identifier.
```
HighlightWhitespace(str: String)
```
Space and line ending characters.

Lexer

opaque </>

pub opaque type Lexer

SigilDelimiter

</>

pub type SigilDelimiter {
  ParensSigil
  BraceSigil
  BracketSigil
  AngleSigil
  DoubleQuotedSigil
  SingleQuotedSigil
  PipeSigil
  SlashSigil
  HeredocSigil
}

Constructors

```
ParensSigil
```
```
BraceSigil
```
```
BracketSigil
```
```
AngleSigil
```
```
DoubleQuotedSigil
```
```
SingleQuotedSigil
```
```
PipeSigil
```
```
SlashSigil
```
```
HeredocSigil
```

StringDelimiter

</>

pub type StringDelimiter {
  SingleQuotedString
  DoubleQuotedString
  HeredocString
}

Constructors

```
SingleQuotedString
```
```
DoubleQuotedString
```
```
HeredocString
```

Token

</>

A token useful for parsing or processing Elixir source code.

pub type Token {
  EndOfLine(str: String)
  Whitespace(str: String)
  InlineComment(line: String)
  Fn
  Do
  End
  Not
  And
  Or
  When
  In
  After
  Else
  Catch
  Rescue
  LessLessLess
  GreaterGreaterGreater
  PipePipePipe
  AmpersandAmpersandAmpersand
  CaretCaretCaret
  TildeTildeTilde
  EqualEqualEqual
  BangEqualEqual
  TildeGreaterGreater
  LessLessTilde
  LessTildeGreater
  PipeTildeGreater
  LessPipeGreater
  PlusPlusPlus
  MinusMinusMinus
  LessLess
  GreaterGreater
  EqualEqual
  BangEqual
  LessEqual
  GreaterEqual
  AmpersandAmpersand
  PipePipe
  BackslashBackslash
  LessGreater
  PlusPlus
  MinusMinus
  PipeGreater
  EqualTilde
  MinusGreater
  LessMinus
  TildeGreater
  LessTilde
  ColonColon
  DotDot
  Equal
  Less
  Greater
  Plus
  Minus
  Asterisk
  Slash
  Pipe
  Dot
  Caret
  Ampersand
  Bang
  SlashSlash
  AsteriskAsterisk
  EqualGreater
  Percent
  OpenParen
  CloseParen
  OpenBracket
  CloseBracket
  OpenBrace
  CloseBrace
  Comma
  Semicolon
  At
  Nil
  True
  False
  Identifier(name: String)
  Alias(name: String)
  KeywordIdentifier(name: String)
  Atom(name: String)
  Integer(base: Base, value: String)
  FloatingPoint(str: String)
  Character(str: String)
  String(
    delimiter: StringDelimiter,
    parts: List(String),
    values: List(List(Token)),
  )
  StringAtom(
    delimiter: StringDelimiter,
    parts: List(String),
    values: List(List(Token)),
  )
  StringKeywordIdentifier(
    delimiter: StringDelimiter,
    parts: List(String),
    values: List(List(Token)),
  )
  Sigil(
    identifier: String,
    delimiter: SigilDelimiter,
    parts: List(String),
    values: List(List(Token)),
    modifiers: String,
  )
  VerbatimSigil(
    identifier: String,
    delimiter: SigilDelimiter,
    value: String,
    modifiers: String,
  )
  Unexpected(String)
  MultiCodepointCharacter(String)
  KeywordIdentifierWithoutSpace(String)
  InvalidIdentifier(String)
  InvalidAlias(String)
  InvalidAtom(String)
  InvalidKeywordIdentifier(String)
  InvalidNumber(String)
  UnterminatedString(
    delimiter: StringDelimiter,
    parts: List(String),
    values: List(List(Token)),
  )
  UnterminatedStringAtom(
    delimiter: StringDelimiter,
    parts: List(String),
    values: List(List(Token)),
  )
  InvalidSigilIdentifier(String)
  UnterminatedVerbatimSigil(
    identifier: String,
    delimiter: SigilDelimiter,
    value: String,
  )
  InvalidVerbatimSigilModifier(
    identifier: String,
    delimiter: SigilDelimiter,
    value: String,
    modifiers: String,
  )
  UnterminatedSigil(
    identifier: String,
    delimiter: SigilDelimiter,
    parts: List(String),
    values: List(List(Token)),
  )
  InvalidSigilModifier(
    identifier: String,
    delimiter: SigilDelimiter,
    parts: List(String),
    values: List(List(Token)),
    modifiers: String,
  )
}

Constructors

```
EndOfLine(str: String)
```
```
Whitespace(str: String)
```
```
InlineComment(line: String)
```
```
Fn
```
```
Do
```
```
End
```
```
Not
```
```
And
```
```
Or
```
```
When
```
```
In
```
```
After
```
```
Else
```
```
Catch
```
```
Rescue
```
```
LessLessLess
```
```
GreaterGreaterGreater
```
```
PipePipePipe
```
```
AmpersandAmpersandAmpersand
```
```
CaretCaretCaret
```
```
TildeTildeTilde
```
```
EqualEqualEqual
```
```
BangEqualEqual
```
```
TildeGreaterGreater
```
```
LessLessTilde
```
```
LessTildeGreater
```
```
PipeTildeGreater
```
```
LessPipeGreater
```
```
PlusPlusPlus
```
```
MinusMinusMinus
```
```
LessLess
```
```
GreaterGreater
```
```
EqualEqual
```
```
BangEqual
```
```
LessEqual
```
```
GreaterEqual
```
```
AmpersandAmpersand
```
```
PipePipe
```
```
BackslashBackslash
```
```
LessGreater
```
```
PlusPlus
```
```
MinusMinus
```
```
PipeGreater
```
```
EqualTilde
```
```
MinusGreater
```
```
LessMinus
```
```
TildeGreater
```
```
LessTilde
```
```
ColonColon
```
```
DotDot
```
```
Equal
```
```
Less
```
```
Greater
```
```
Plus
```
```
Minus
```
```
Asterisk
```
```
Slash
```
```
Pipe
```
```
Dot
```
```
Caret
```
```
Ampersand
```
```
Bang
```
```
SlashSlash
```
```
AsteriskAsterisk
```
```
EqualGreater
```
```
Percent
```
```
OpenParen
```
```
CloseParen
```
```
OpenBracket
```
```
CloseBracket
```
```
OpenBrace
```
```
CloseBrace
```
```
Comma
```
```
Semicolon
```
```
At
```
```
Nil
```
```
True
```
```
False
```
```
Identifier(name: String)
```
```
Alias(name: String)
```
```
KeywordIdentifier(name: String)
```
```
Atom(name: String)
```
```
Integer(base: Base, value: String)
```
```
FloatingPoint(str: String)
```
```
Character(str: String)
```

String(
  delimiter: StringDelimiter,
  parts: List(String),
  values: List(List(Token)),
)

StringAtom(
  delimiter: StringDelimiter,
  parts: List(String),
  values: List(List(Token)),
)

StringKeywordIdentifier(
  delimiter: StringDelimiter,
  parts: List(String),
  values: List(List(Token)),
)

Sigil(
  identifier: String,
  delimiter: SigilDelimiter,
  parts: List(String),
  values: List(List(Token)),
  modifiers: String,
)

VerbatimSigil(
  identifier: String,
  delimiter: SigilDelimiter,
  value: String,
  modifiers: String,
)

```
Unexpected(String)
```
The parser got into a state where no character could be consumed. The Unexpected token will contain all graphemes up to the point where the parser could continue successfully.
```
MultiCodepointCharacter(String)
```
Got a grapheme cluster with more than one codepoint following a character literal (?X).
```
KeywordIdentifierWithoutSpace(String)
```
A keyword identifier must be followed by a space.
```
InvalidIdentifier(String)
```
An invalid identifier. Right now this only detects identifiers containing an at (@) symbol.
```
InvalidAlias(String)
```
```
InvalidAtom(String)
```
An invalid atom. A normal (non-quoted) cannot start with a number.
```
InvalidKeywordIdentifier(String)
```
An invalid keyword identifier. Right now this token is never produced.
```
InvalidNumber(String)
```
An invalid number literal.

UnterminatedString(
  delimiter: StringDelimiter,
  parts: List(String),
  values: List(List(Token)),
)

A string that did not end properly before the end of the file.

UnterminatedStringAtom(
  delimiter: StringDelimiter,
  parts: List(String),
  values: List(List(Token)),
)

A string atom that did not end properly before the end of the file.

```
InvalidSigilIdentifier(String)
```
An invalid sigil name. Sigils have to either be a single ascii lowercase letter or an ascii uppercase letter followed by one or more letters or digits.

UnterminatedVerbatimSigil(
  identifier: String,
  delimiter: SigilDelimiter,
  value: String,
)

A verbatim sigil did not end properly before the end of the file.

InvalidVerbatimSigilModifier(
  identifier: String,
  delimiter: SigilDelimiter,
  value: String,
  modifiers: String,
)

A verbatim sigil followed by an invalid modifier. Only ASCII letters and digits are allowed as modifiers.

UnterminatedSigil(
  identifier: String,
  delimiter: SigilDelimiter,
  parts: List(String),
  values: List(List(Token)),
)

A sigil did not end properly before the end of the file.

InvalidSigilModifier(
  identifier: String,
  delimiter: SigilDelimiter,
  parts: List(String),
  values: List(List(Token)),
  modifiers: String,
)

A sigil followed by an invalid modifier. Only ASCII letters and digits are allowed as modifiers.

Values

highlight

</>

pub fn highlight(source: String) -> List(HighlightToken)

Parse Elixir source code into a list of HighlightToken using default settings.

The resulting token list can be highlighted by using functions like to_ansi, to_html, or by pattern matching on them youself.

ignore_inline_comments

</>

pub fn ignore_inline_comments(lexer: Lexer) -> Lexer

Skip inline comment tokens in the resulting token list.

Note that in Elixir, comments are often added using attributes, which will still be included.

ignore_whitespace

</>

pub fn ignore_whitespace(lexer: Lexer) -> Lexer

Skip whitespace and end-of-line tokens in the resulting token list.

new

</>

pub fn new() -> Lexer

Create a new Lexer with default settings.

Lexers can be cached and reused to parse many source files efficiently.

to_ansi

</>

pub fn to_ansi(tokens: List(HighlightToken)) -> String

Format a list of HighlightToken into ansi highlighting for display in a terminal.

Colours based on contour:

Token	Colour
Alias, Atom	Cyan
Attribute, Keyword	Yellow
Builtin	Bright Yellow
Function	Blue
Operator	Magenta
String, Number, Literal	Green
Error	Red Background
Whitespace, Variable	No colour
Comment	Italic grey

to_highlight

</>

pub fn to_highlight(tokens: List(Token)) -> List(HighlightToken)

Convert a list of lexer tokens into a list of HighlightTokens.

to_html

</>

pub fn to_html(tokens: List(HighlightToken)) -> String

Format a list of HighlightToken as HTML. Each token is wrapped inside a <span> tag with a class indicating the type.

Class names are based on contour:

Token	CSS class
Alias	hl-module
Atom	hl-symbol
Attribute	hl-attribute
Builtin	hl-built_in
Comment	hl-comment
Function	hl-function
Identifier	hl-variable
Keyword	hl-keyword
Number	hl-number
String	hl-string
Literal	hl-literal
Operator	hl-operator
Punctuation	hl-punctuation
Error	hl-error
Whitespace	no class

Place the output within a <pre><code class="language-elixir">...</code></pre> and add styling for these CSS classes to get highlighting on your website. Here’s some CSS you could use:

pre code {
  .hl-comment   { color: #d4d4d4; font-style: italic }
  .hl-function  { color: #9ce7ff }
  .hl-keyword   { color: #ffd596 }
  .hl-built_in  { color: #ffd596 }
  .hl-attribute { color: #ffd596 }
  .hl-operator  { color: #ffaff3 }
  .hl-string    { color: #c8ffa7 }
  .hl-number    { color: #c8ffa7 }
  .hl-litreal   { color: #c8ffa7 }
  .hl-module    { color: #ffddfa }
  .hl-error     { background: red; color: white; }
}

to_source

</>

pub fn to_source(tokens: List(Token)) -> String

Convert a list of tokens back into their source representation.

tokenise

</>

pub fn tokenise(lexer: Lexer, input: String) -> List(Token)

Parse Elixir source code into a list of lexer tokens.