Normandy.Context.WindowManager (normandy v0.2.0)

View Source

Manages context window limits for conversations with automatic truncation strategies.

Provides utilities for tracking token usage, managing context window limits, and automatically truncating conversation history when approaching limits.

Features

  • Token counting for messages and conversations
  • Configurable context window limits per model
  • Multiple truncation strategies (oldest-first, sliding window, summarization)
  • Automatic context management with AgentMemory integration

Example

# Create a window manager with 100k token limit
manager = WindowManager.new(max_tokens: 100_000)

# Check if we're within limits
{:ok, within_limit?} = WindowManager.within_limit?(manager, agent)

# Automatically truncate if needed
{:ok, updated_agent} = WindowManager.ensure_within_limit(agent, manager)

Truncation Strategies

  • :oldest_first - Remove oldest messages first (default)
  • :sliding_window - Keep most recent N messages
  • :summarize - Summarize old messages before removing

Summary

Types

strategy()
t()

Functions

ensure_within_limit(agent, manager)

Ensures agent's conversation stays within context window limit.

estimate_conversation_tokens(memory)

Estimates total tokens in conversation history.

estimate_message_content_tokens(content)

Estimates token count for message content.

estimate_tokens(text)

Estimates token count for a message.

for_model(model, opts \\ [])

Creates a WindowManager configured for a specific model.

new(opts \\ [])

Creates a new WindowManager with optional configuration.

truncate_conversation(agent, manager)

Truncates conversation history using the configured strategy.

within_limit?(manager, agent)

Checks if current conversation is within token limit.

Types

strategy()

@type strategy() :: :oldest_first | :sliding_window | :summarize

t()

@type t() :: %Normandy.Context.WindowManager{
  max_tokens: pos_integer(),
  model_limits: map(),
  reserved_tokens: pos_integer(),
  strategy: strategy()
}

Functions

ensure_within_limit(agent, manager)

@spec ensure_within_limit(
  struct(),
  t()
) :: {:ok, struct()}

Ensures agent's conversation stays within context window limit.

Automatically truncates history if needed using the configured strategy.

Example

{:ok, updated_agent} = WindowManager.ensure_within_limit(agent, manager)

estimate_conversation_tokens(memory)

@spec estimate_conversation_tokens(map()) :: pos_integer()

Estimates total tokens in conversation history.

Example

tokens = WindowManager.estimate_conversation_tokens(agent.memory)

estimate_message_content_tokens(content)

@spec estimate_message_content_tokens(term()) :: pos_integer()

Estimates token count for message content.

Example

tokens = WindowManager.estimate_message_content_tokens("Hello")
#=> ~2

estimate_tokens(text)

@spec estimate_tokens(String.t()) :: pos_integer()

Estimates token count for a message.

This is a rough estimate based on character count. For accurate counts, use count_tokens_api/2.

Example

tokens = WindowManager.estimate_tokens("Hello, world!")
#=> ~4 tokens

for_model(model, opts \\ [])

@spec for_model(
  String.t(),
  keyword()
) :: t()

Creates a WindowManager configured for a specific model.

Uses the model's known context window limit.

Example

manager = WindowManager.for_model("claude-3-5-sonnet-20241022")

new(opts \\ [])

@spec new(keyword()) :: t()

Creates a new WindowManager with optional configuration.

Options

  • :max_tokens - Maximum tokens for context window (default: 100,000)
  • :reserved_tokens - Tokens to reserve for response (default: 4096)
  • :strategy - Truncation strategy (default: :oldest_first)

Example

manager = WindowManager.new(max_tokens: 50_000, strategy: :sliding_window)

truncate_conversation(agent, manager)

@spec truncate_conversation(
  struct(),
  t()
) :: {:ok, struct()}

Truncates conversation history using the configured strategy.

Example

{:ok, updated_agent} = WindowManager.truncate_conversation(agent, manager)

within_limit?(manager, agent)

@spec within_limit?(
  t(),
  struct()
) :: {:ok, boolean()}

Checks if current conversation is within token limit.

Example

case WindowManager.within_limit?(manager, agent) do
  {:ok, true} -> :continue
  {:ok, false} -> :truncate_needed
end