tear

tear is a lexer/highlighter for Elixir. It faithfully parses Elixir into a list of tokens similar to the tokeniser inside the Elixir compiler. It passes all applicatable tests in the Elixir source tree.

Additionally, it provides APIs heavily inspired by contour to highlight Elixir code in HTML or in the terminal.

Example

gleam add tear@1

import gleam/io
import simplifile
import tear

pub fn main() {
  let assert Ok(input) = simplifile.read("/dev/stdin")

  io.println(
    input
    |> tear.highlight
    |> tear.to_ansi
  )
}

Differences to Elixir

Elixirs lexer enforces additional rules w.r.t. identifers; Only certain (Unicode) classes of characters are allowed, you cannot mix scripts within one “section”, confusable characters are forbidden, and all identifiers are normalised to NFC form. tear does not implement those rules - instead, identifiers may contain any sequence of graphemes that is not otherwise significant to the parser. This may be improved upon in the future.

Elixir enforces some other parsing rules at the lexer level - for example, it tracks nesting parenthesis and do blocks, and enforces a rule that no space is allowed between functions and the argument list. The main focus of tear is to produce a token stream for syntax highlighting, so it instead opts for accepting those things in the lexer instead.

The format of heredoc strings is not validated or changed by the lexer.