View Source TextParser.Token behaviour (text_parser v0.1.0)

Behaviour for implementing custom token extractors.

Example

defmodule MyApp.CustomToken do
  use TextParser.Token,
    pattern: ~r/(?:^| )(@S*)/u,
    trim_chars: [",", ".", "!", "?"]

  def is_valid?(token) do
    # implement validation logic
  end
end

Summary

Callbacks

extract(text)

Callback for extracting tokens from text.

is_valid?(token)

Callback for validating a token value.

Functions

__using__(opts)

When used, defines the token struct and implements the behaviour.

Callbacks

extract(text)

@callback extract(text :: String.t()) :: [struct()]

Callback for extracting tokens from text.

The implementation should return a list of token structs, where each token has a :value and :position field. The position should be a tuple of {start_pos, end_pos} indicating the token's position in the text.

is_valid?(token)

@callback is_valid?(token :: term()) :: boolean()

Callback for validating a token value.

This should be implemented by each token module to define its specific validation rules.

Functions

using(opts)

(macro)

When used, defines the token struct and implements the behaviour.

Options

:pattern - Required. A regex pattern to match tokens in text
:trim_chars - Optional. A list of characters to trim from the end of matched tokens