Produces a classified token stream from an MF2 AST for syntax highlighting.
The output is a flat list of {class :: atom, text :: String.t()}
tuples. Each tuple represents one syntactic region of the canonical
message — punctuation, a variable reference, a function name, a
literal, plain text, etc. Formatters (Localize.Message.Formatter.HTML,
Localize.Message.Formatter.ANSI, Localize.Message.Formatter.Plain)
consume the token stream and produce rendered output.
Concatenating every token's text field yields the canonical MF2
message (the same output as Localize.Message.Print.to_string/1) —
the Plain formatter does exactly that. This property is verified
by the test suite.
Token classes
Classes follow the same taxonomy as tree-sitter highlight captures,
so one stylesheet can style both server-rendered HTML and the
browser-side mf2_wasm_editor.
Atoms carry underscores; the HTML formatter converts them to hyphens
on emission (so :punctuation_bracket becomes .mf2-punctuation-bracket).
:punctuation_bracket—{,},{{,}},|,=,/, whitespace inside expressions, and similar structural characters.:variable—$name(entire token including$).:function—:number,:date, custom function names (entire token including:).:keyword—.input,.local,.matchkeywords.:tag— markup tag names (boldin{#bold}).:attribute— attribute names (@translate).:property— option names (styleinstyle=|short|).:string— quoted literal content including the|delimiters.:number— numeric literals (integer and float).:text— pattern text (plain message content).:string_escape—\{,\|,\\escape sequences.:constant_builtin—*(catchall variant key).
Summary
Functions
Walks the AST and returns a list of classified tokens.
Types
Functions
Walks the AST and returns a list of classified tokens.
Arguments
astis a parsed MF2 AST as returned byLocalize.Message.Parser.parse/1.
Returns
- A list of
{class, text}tuples.