Rag.Chunker.Character (rag v0.3.4)

View Source

Fixed-size chunking with overlap and smart boundaries.

Attempts to break at sentence boundaries, falls back to word boundaries, then to exact character positions.

Options

  • max_chars - Maximum characters per chunk (default: 500)
  • overlap - Characters to overlap between chunks (default: 50)

Summary

Types

t()

Functions

chunk(chunker, text, opts)

Split text into character-based chunks.

default_opts()

Returns default options for the character chunker.

Types

t()

@type t() :: %Rag.Chunker.Character{
  max_chars: pos_integer(),
  overlap: non_neg_integer()
}

Functions

chunk(chunker, text, opts)

@spec chunk(t(), String.t(), keyword()) :: [Rag.Chunker.Chunk.t()]

Split text into character-based chunks.

default_opts()

@spec default_opts() :: keyword()

Returns default options for the character chunker.