Sifter.Query.Lexer.State (Sifter v0.2.0)

View Source

Internal lexer state for tracking position during string scanning.

This module maintains the lexer's position within the source string and accumulates tokens as they are recognized during the scanning process.

Fields

  • src - Original complete source string
  • rest - Remaining unprocessed bytes
  • off - Current byte offset in the source
  • len - Total length of the source string
  • acc - Accumulated tokens (in reverse order)
  • prev_term? - Whether the previous token was a term (affects implied AND insertion)

Used internally by the lexer's scanning functions.

Summary

Types

t()

Lexer state for tracking position during string scanning.

Types

t()

@type t() :: %Sifter.Query.Lexer.State{
  acc: [Sifter.Query.Lexer.token()],
  len: non_neg_integer(),
  off: non_neg_integer(),
  prev_term?: boolean(),
  rest: binary(),
  src: binary()
}

Lexer state for tracking position during string scanning.

  • src: Original complete source string
  • rest: Remaining bytes to scan
  • off: Current byte offset in source
  • len: Total source length in bytes
  • acc: Accumulated tokens (reversed)
  • prev_term?: True after term tokens, false after structural tokens