# `CommBus.Tokenizer`
[🔗](https://github.com/fosferon/comm_bus/blob/v0.1.0/{path}#L{line})

Token counting façade with pluggable backends.

Configure with:

    config :comm_bus, :tokenizer, CommBus.Tokenizer.Simple

or supply `tokenizer: MyTokenizer` in function opts.

# `count_message`
[🔗](https://github.com/fosferon/comm_bus/blob/v0.1.0/{path}#L{line})

```elixir
@callback count_message(
  CommBus.Message.t(),
  keyword()
) :: non_neg_integer()
```

# `count_tokens`
[🔗](https://github.com/fosferon/comm_bus/blob/v0.1.0/{path}#L{line})

```elixir
@callback count_tokens(
  String.t(),
  keyword()
) :: non_neg_integer()
```

# `annotate_entries`
[🔗](https://github.com/fosferon/comm_bus/blob/v0.1.0/{path}#L{line})

```elixir
@spec annotate_entries(
  [CommBus.Entry.t()],
  keyword()
) :: [CommBus.Entry.t()]
```

Annotates a list of entries with token counts by calling `annotate_entry/2`
on each entry.

## Parameters

  - `entries` — List of `%CommBus.Entry{}` structs.
  - `opts` — Keyword options forwarded to the tokenizer backend.

## Returns

A list of entries with `token_count` fields populated.

# `annotate_entry`
[🔗](https://github.com/fosferon/comm_bus/blob/v0.1.0/{path}#L{line})

```elixir
@spec annotate_entry(
  CommBus.Entry.t(),
  keyword()
) :: CommBus.Entry.t()
```

Fills in the `token_count` field of an entry by counting tokens in its content.

If the entry already has a non-nil `token_count`, it is returned unchanged.

## Parameters

  - `entry` — A `%CommBus.Entry{}` struct.
  - `opts` — Keyword options forwarded to the tokenizer backend.

## Returns

The entry with `token_count` populated.

# `message_count`
[🔗](https://github.com/fosferon/comm_bus/blob/v0.1.0/{path}#L{line})

```elixir
@spec message_count(
  CommBus.Message.t(),
  keyword()
) :: non_neg_integer()
```

Counts the number of tokens in a message, including role overhead.

## Parameters

  - `message` — A `%CommBus.Message{}` struct.
  - `opts` — Keyword options; `:tokenizer` overrides the configured backend.

## Returns

A non-negative integer representing the total token count for the message.

# `token_count`
[🔗](https://github.com/fosferon/comm_bus/blob/v0.1.0/{path}#L{line})

```elixir
@spec token_count(
  String.t(),
  keyword()
) :: non_neg_integer()
```

Counts the number of tokens in the given text string using the configured
tokenizer backend.

## Parameters

  - `text` — The text string to tokenize.
  - `opts` — Keyword options; `:tokenizer` overrides the configured backend.

## Returns

A non-negative integer representing the token count.

# `tokenizer`
[🔗](https://github.com/fosferon/comm_bus/blob/v0.1.0/{path}#L{line})

```elixir
@spec tokenizer(keyword()) :: module()
```

Returns the tokenizer module resolved from options or application config.

Checks, in order: the `:tokenizer` key in `opts`, the `:tokenizer` application
env for `:comm_bus`, and falls back to `CommBus.Tokenizer.Simple`.

## Parameters

  - `opts` — Keyword options; `:tokenizer` overrides the configured backend.

## Returns

The tokenizer module (an atom implementing the `CommBus.Tokenizer` behaviour).

---

*Consult [api-reference.md](api-reference.md) for complete listing*