PtcRunner.SubAgent.Loop.Metrics (PtcRunner v0.9.0)

Copy Markdown View Source

Telemetry, tracing, and usage metrics for SubAgent execution.

This module handles:

  • Token accumulation across LLM calls
  • Final usage statistics (duration, memory, turns, tokens)
  • Trace entry construction with optional debug info
  • Turn struct construction for execution history
  • Trace filtering based on execution result

Summary

Functions

Accumulate tokens from an LLM call into state.

Apply trace filtering based on trace_mode and execution result.

Build final usage map with token counts from accumulated state.

Build a truncated preview of the result for telemetry metadata.

Build token measurements map for telemetry.

Build a Turn struct for the current execution cycle.

Build measurements for turn stop event with optional tokens.

Emit turn stop event immediately after a turn completes.

Estimate token count for a text string.

Extract program from the last turn in a result.

Functions

accumulate_tokens(state, tokens)

@spec accumulate_tokens(map(), map() | nil) :: map()

Accumulate tokens from an LLM call into state.

Parameters

  • state - Current loop state
  • tokens - Token counts map with :input, :output, :cache_creation, :cache_read keys, or nil

Returns

Updated state with accumulated token counts.

apply_trace_filter(trace, trace_mode, is_error)

@spec apply_trace_filter(list() | nil, boolean() | :on_error, boolean()) ::
  list() | nil

Apply trace filtering based on trace_mode and execution result.

Filter Modes

  • true - Always include trace
  • false - Never include trace (returns nil)
  • :on_error - Include trace only when is_error is true

build_final_usage(state, duration_ms, memory_bytes, turn_offset \\ 0)

@spec build_final_usage(map(), non_neg_integer(), non_neg_integer(), integer()) ::
  map()

Build final usage map with token counts from accumulated state.

Parameters

  • state - Current loop state with accumulated metrics
  • duration_ms - Total execution duration in milliseconds
  • memory_bytes - Memory used in bytes
  • turn_offset - Offset for turn count (0 for completed turns, -1 for pre-turn failures)

Returns

Map with usage statistics including cache token metrics and compression stats when available.

build_result_preview(result)

@spec build_result_preview(term()) :: String.t()

Build a truncated preview of the result for telemetry metadata.

Truncates to 65536 characters.

build_token_measurements(tokens)

@spec build_token_measurements(map() | nil) :: map()

Build token measurements map for telemetry.

build_turn(state, raw_response, program, result, opts \\ [])

@spec build_turn(map(), String.t(), String.t() | nil, term(), keyword()) ::
  PtcRunner.Turn.t()

Build a Turn struct for the current execution cycle.

Creates either a success or failure Turn based on the success? option.

Parameters

  • state - Current loop state (used for turn number and messages)
  • raw_response - Full LLM response text
  • program - PTC-Lisp program that was executed (or nil if parsing failed)
  • result - Execution result or error
  • opts - Keyword options:
    • success? - Whether this turn succeeded (default: true)
    • prints - Captured println output (default: [])
    • tool_calls - Tool invocations made during this turn (default: [])
    • memory - Memory state after this turn (default: state.memory)
    • type - Turn type: :normal, :must_return, or :retry (default: :normal)

Returns

A %Turn{} struct.

build_turn_measurements(duration, tokens)

@spec build_turn_measurements(integer(), map() | nil) :: map()

Build measurements for turn stop event with optional tokens.

emit_turn_stop_immediate(turn, agent, state, turn_start, turn_tokens \\ nil)

@spec emit_turn_stop_immediate(
  PtcRunner.Turn.t() | nil,
  PtcRunner.SubAgent.t() | map(),
  map(),
  integer(),
  map() | nil
) :: :ok

Emit turn stop event immediately after a turn completes.

This is used by the iterative driver_loop to emit telemetry right after each turn, rather than batching events when the stack unwinds. Handles nil turn defensively for cases where LLM errors occur before a Turn struct is created.

Parameters

  • turn - The Turn struct for this turn, or nil if LLM error occurred before turn creation
  • agent - The SubAgent struct
  • state - Current loop state
  • turn_start - Monotonic timestamp when turn started
  • turn_tokens - Optional token counts from LLM call (overrides state.turn_tokens if provided)

estimate_tokens(text)

@spec estimate_tokens(String.t() | nil) :: non_neg_integer()

Estimate token count for a text string.

Uses a simple approximation of ~4 characters per token, which is reasonably accurate for most LLM tokenizers (within ~10-20%).

Examples

iex> PtcRunner.SubAgent.Loop.Metrics.estimate_tokens("Hello world")
2

iex> PtcRunner.SubAgent.Loop.Metrics.estimate_tokens("")
0

iex> PtcRunner.SubAgent.Loop.Metrics.estimate_tokens(nil)
0

extract_program_from_result(arg1)

@spec extract_program_from_result(tuple()) :: String.t() | nil

Extract program from the last turn in a result.

The result is {:ok, step} or {:error, step} for final results. For continuation results (loop), this returns nil.

Note: step.turns is in chronological order (first turn first, last turn last).

Examples

iex> step = %PtcRunner.Step{turns: [%{program: "code"}]}
iex> PtcRunner.SubAgent.Loop.Metrics.extract_program_from_result({:ok, step})
"code"

iex> PtcRunner.SubAgent.Loop.Metrics.extract_program_from_result({:ok, %PtcRunner.Step{turns: []}})
nil

iex> PtcRunner.SubAgent.Loop.Metrics.extract_program_from_result({:error, :invalid})
nil