View Source TextChunker.Strategies.RecursiveChunk.Separators (TextChunker v0.3.1)

Handles separator configuration for the RecursiveChunk text chunking strategy.

Provides predefined lists of separators tailored for different text formats. The order of separators is crucial, as the splitting algorithm prioritizes them sequentially.

Key Features:

  • Format-Specific Separators: Customizes splitting behavior based on formats like Markdown, plain text, Elixir, and others.
  • Prioritized Splitting: Attempts to split text using the highest-priority separator applicable to the text's content.

Summary

Functions

Returns a list of separators that will be used to split the document of the given format

Functions

@spec get_separators(atom()) :: [String.t()]

Returns a list of separators that will be used to split the document of the given format