Rag.Chunker behaviour (rag v0.3.4)

Behaviour for text chunking strategies.

Chunkers split text into smaller pieces suitable for embedding and retrieval. Each chunk includes byte positions for source reconstruction.

Summary

t()

Split text into chunks.

default_opts()

Returns default options for this chunker.

Dispatch to the chunker implementation.

Chunk an ingestion map, adding chunks to the result.

@type t() :: struct()

@callback chunk(chunker :: t(), text :: String.t(), opts :: keyword()) :: [
  Rag.Chunker.Chunk.t()
]

Split text into chunks.

Returns a list of Rag.Chunker.Chunk structs.

(optional)

@callback default_opts() :: keyword()

Returns default options for this chunker.

@spec chunk(t(), String.t(), keyword()) :: [Rag.Chunker.Chunk.t()]

Dispatch to the chunker implementation.

@spec chunk_ingestion(t(), map(), keyword()) :: map()

Chunk an ingestion map, adding chunks to the result.

Expects input map with a :document key containing text. Returns a map with :chunks added.