ExNlp.Token (ex_nlp v0.1.0)

View Source

Token struct representing a single token in text.

A token contains the text content along with optional position and offset information for tracking the token's location in the original text.

Fields

  • text (required): The token text content
  • position (optional, default: 0): The token's position in the tokenized sequence
  • start_offset (optional, default: 0): The starting byte offset in the original text
  • end_offset (optional, default: 0): The ending byte offset in the original text

Examples

# Create a token with just text
iex> %ExNlp.Token{text: "hello"}
%ExNlp.Token{text: "hello", position: 0, start_offset: 0, end_offset: 0}

# Create a token with all fields
iex> %ExNlp.Token{text: "hello", position: 1, start_offset: 6, end_offset: 11}
%ExNlp.Token{text: "hello", position: 1, start_offset: 6, end_offset: 11}

# Pattern matching
iex> token = %ExNlp.Token{text: "hello", position: 0}
iex> token.text
"hello"

Summary

Types

t()

Functions

from_map(map)

Creates a token from a map (for backward compatibility during migration).

to_map(token)

Converts a token to a map (for backward compatibility if needed).

Types

t()

@type t() :: %ExNlp.Token{
  end_offset: non_neg_integer(),
  position: non_neg_integer(),
  start_offset: non_neg_integer(),
  text: String.t()
}

Functions

from_map(map)

@spec from_map(map()) :: t()

Creates a token from a map (for backward compatibility during migration).

Allows creating a token from maps that may have :text and optionally position/offset fields.

Examples

iex> ExNlp.Token.from_map(%{text: "hello"})
%ExNlp.Token{text: "hello", position: 0, start_offset: 0, end_offset: 0}

iex> ExNlp.Token.from_map(%{text: "hello", position: 1})
%ExNlp.Token{text: "hello", position: 1, start_offset: 0, end_offset: 0}

to_map(token)

@spec to_map(t()) :: map()

Converts a token to a map (for backward compatibility if needed).

Examples

iex> token = %ExNlp.Token{text: "hello", position: 1}
iex> ExNlp.Token.to_map(token)
%{text: "hello", position: 1, start_offset: 0, end_offset: 0}