View Source Makeup.Lexer.Combinators (Makeup v1.2.1)

Common components useful in many lexers.

Summary

Functions

Joins the result of the given combinator into a single string.

Matches a given combinator, repeated 0 or more times, surrounded by left and right delimiters.

A generic combinator for string-like syntactic structures.

Wraps the given combinator into a token of the given ttype.

Matches one of the literal strings in the list.

Matches one of the literal strings in the list and wraps it in a token of the given type.

Matches one of the literal strings in the list and wraps it in a token of the given type, with the given attrs.

Functions

Joins the result of the given combinator into a single string.

This is not usually necessary, but it can be useful if you want to match on the tokens. It's easier to match on the token {:keyword, %{}, "unquote"} than on something like {:keyword, %{}, ["u", "nquote"]}, even though both tokens will be treated the same way by the formatter.

Link to this function

many_surrounded_by(combinator, left, right, opts \\ [])

View Source

Matches a given combinator, repeated 0 or more times, surrounded by left and right delimiters.

Delimiters can be combinators or literal strings (either both combinators or both literal strings). In case of literal strings, this function wraps the right and left delimiters into a token of the ttype option (default: :punctuation).

It also succeeds if we get to the end of string and the closing delimiter is missing, to avoid parsing the program multiple times in case of mismatched delimeters or invalid programs. This behavior can be disabled by passing eos: false as option.

Link to this function

string_like(left, right, middle, ttype, attrs \\ %{})

View Source

A generic combinator for string-like syntactic structures.

It takes the following parameters:

  • left - left delimiter for the string. Can be a binary or a general combinator.
  • right - right delimiter for the string. Can be a binary or a general combinator
  • middle - a list of parsers to run inside the string which parse entities that aren't characters. The most common example are special characters and string interpolation for languages that support it like Elixir.
  • ttype - the token type to use for the string delimiters and ordinary characters (tokens parsd by the )
  • attrs - metadata attributes for the string delimiters and ordinary characters

Examples

single_quoted_heredocs = string_like(
  "'''",
  "'''",
  combinators_inside_string,
  :string_char
)

The above is equivalent to the following more explicit version:

single_quoted_heredocs = string_like(
  string("'''"),
  string("'''"),
  combinators_inside_string,
  :string_char
)
Link to this function

token(literal, token_type)

View Source

Wraps the given combinator into a token of the given ttype.

Instead of a combinator, the first argument can also be a string literal.

Link to this function

token(literal, token_type, attrs)

View Source

Matches one of the literal strings in the list.

The strings aren't matched in order: they are automatically sorted in a way that guarantees that the longest strings will be tried first.

Examples

keywords = word_from_list(~w[do end catch after rescue])
Link to this function

word_from_list(words, ttype)

View Source

Matches one of the literal strings in the list and wraps it in a token of the given type.

This is is just a shorthand.

The strings aren't matched in order: they are automatically sorted in a way that guarantees that the longest strings will be tried first.

Examples

keywords = word_from_list(~w[do end catch after rescue], :keyword)
Link to this function

word_from_list(words, ttype, attrs)

View Source

Matches one of the literal strings in the list and wraps it in a token of the given type, with the given attrs.

This is is just a shorthand.

The strings aren't matched in order: they are automatically sorted in a way that guarantees that the longest strings will be tried first.