sqlode/lexer
Types
A SQL token produced by the lexer.
pub type Token {
Keyword(String)
Ident(String)
QuotedIdent(String)
StringLit(String)
NumberLit(String)
Placeholder(String)
Operator(String)
LParen
RParen
Comma
Semicolon
Dot
Star
}
Constructors
-
Keyword(String)SQL keyword (lowercased): SELECT, FROM, CREATE, etc.
-
Ident(String)Unquoted identifier: table_name, column_name
-
QuotedIdent(String)Quoted identifier: “name”,
name, [name] (quotes stripped) -
StringLit(String)String literal (quotes stripped): ‘value’, dollar
-
NumberLit(String)Numeric literal: 42, 3.14
-
Placeholder(String)Parameter placeholder: $1, ?, :name, @name
-
Operator(String)Operator: =, <>, ::, ->, ||, etc.
-
LParen -
RParen -
Comma -
Semicolon -
Dot -
Star
Options for controlling how tokens are rendered back to text.
pub type TokenRenderOptions {
TokenRenderOptions(
uppercase_keywords: Bool,
preserve_quotes: Bool,
engine: option.Option(model.Engine),
)
}
Constructors
-
TokenRenderOptions( uppercase_keywords: Bool, preserve_quotes: Bool, engine: option.Option(model.Engine), )Arguments
- uppercase_keywords
-
When True, SQL keywords are rendered in UPPERCASE.
- preserve_quotes
-
When True, quoted identifiers keep their quotes and string literals escape embedded single-quotes.
- engine
-
When set, use engine-specific quote style for identifiers: MySQL → backticks, PostgreSQL/SQLite → double quotes.
Values
pub fn tokenize(sql: String, engine: model.Engine) -> List(Token)
Tokenize a SQL string into a list of tokens. Comments are stripped. String literals and quoted identifiers are preserved as single tokens with their content.
pub fn tokens_to_string(
tokens: List(Token),
options: TokenRenderOptions,
) -> String
Render a list of tokens back to a SQL string with smart spacing.