@callback chunk(
  text :: String.t(),
  tokenizer :: Tokenizers.Tokenizer.t(),
  opts :: keyword()
) ::
  {:ok, [Chunx.Chunk.t()]} | {:error, any()}

Splits text into chunks using the given tokenizer.

Parameters

• text - The text to chunk
• tokenizer - The tokenizer to use
• opts - Options specific to the chunking strategy