Rag.Chunker.Semantic (rag v0.3.4)

Semantic chunking using embedding similarity.

Groups sentences based on embedding similarity. Starts a new chunk when similarity drops below threshold or max_chars is reached.

Options

embedding_fn - Function (String.t() -> [float()]) to generate embeddings (required)
threshold - Similarity threshold for grouping (default: 0.8)
max_chars - Maximum characters per chunk (default: 500)

embedding()

t()

Split text into semantic chunks using embedding similarity.

default_opts()

Returns default options for the semantic chunker.

@type embedding() :: [float()]

@type t() :: %Rag.Chunker.Semantic{
  embedding_fn: (String.t() -> embedding()) | nil,
  max_chars: pos_integer(),
  threshold: float()
}

@spec chunk(t(), String.t(), keyword()) :: [Rag.Chunker.Chunk.t()]

Split text into semantic chunks using embedding similarity.

@spec default_opts() :: keyword()

Returns default options for the semantic chunker.