Snakepit (LLM worker pool) integration for semantic measurements.

Snakepit provides:

• LLM-as-judge for quality scoring
• Semantic coherence analysis
• Factuality checking
• Bias detection

Configuration

config :forge, :snakepit,
  url: "http://snakepit.example.com",
  api_key: "your-key",
  default_model: "claude-3-5-sonnet"

Example Usage

defmodule CoherenceMeasurement do
  use Forge.Measurement

  def compute(samples) do
    sample = hd(samples)
    text = sample.data["text"]

    prompt = """
    Rate the semantic coherence of this text on a scale of 1-10.
    Text: #{text}
    """

    {:ok, job_id} = Snakepit.enqueue(prompt, model: "claude-3-5-sonnet")

    case Snakepit.poll_result(job_id, timeout: 20_000) do
      {:ok, response} ->
        score = extract_score(response["body"])
        {:ok, %{score: score, reasoning: response["body"]}}

      {:error, reason} ->
        {:error, reason}
    end
  end
end