Anvil (Anvil v0.1.1)

View Source

Labeling queue library for managing human labeling workflows.

Anvil is a domain-agnostic system for orchestrating human annotation tasks across any sample type - images, text, audio, video, or custom data structures.

Quick Start

# 1. Define a schema
schema = Anvil.Schema.new(
  name: "sentiment",
  fields: [
    %Anvil.Schema.Field{
      name: "sentiment",
      type: :select,
      required: true,
      options: ["positive", "negative", "neutral"]
    }
  ]
)

# 2. Create a queue
{:ok, queue} = Anvil.create_queue(
  queue_id: "sentiment_queue",
  schema: schema
)

# 3. Add samples
Anvil.add_samples(queue, [
  %{id: "s1", text: "Great product!"},
  %{id: "s2", text: "Not good."}
])

# 4. Add labelers
Anvil.add_labelers(queue, ["labeler_1", "labeler_2"])

# 5. Get assignment
{:ok, assignment} = Anvil.get_next_assignment(queue, "labeler_1")

# 6. Submit label
{:ok, label} = Anvil.submit_label(queue, assignment.id, %{"sentiment" => "positive"})

Summary

Functions

add_labelers(queue, labelers)

Adds labelers to a queue.

add_samples(queue, samples)

Adds samples to a queue for labeling.

compute_agreement(queue, opts \\ [])

Computes inter-rater agreement metrics.

create_queue(opts)

Creates a new labeling queue.

export(queue, opts)

Exports labeled data.

get_next_assignment(queue, labeler_id)

Gets the next assignment for a labeler.

skip_assignment(queue, assignment_id, opts \\ [])

Skips an assignment.

submit_label(queue, assignment_id, values)

Submits a label for an assignment.

Functions

add_labelers(queue, labelers)

@spec add_labelers(pid() | atom(), [String.t()]) :: :ok

Adds labelers to a queue.

add_samples(queue, samples)

@spec add_samples(pid() | atom(), [map()]) :: :ok | {:error, term()}

Adds samples to a queue for labeling.

compute_agreement(queue, opts \\ [])

@spec compute_agreement(
  pid() | atom(),
  keyword()
) :: {:ok, float()} | {:error, term()}

Computes inter-rater agreement metrics.

create_queue(opts)

@spec create_queue(keyword()) :: {:ok, pid()} | {:error, term()}

Creates a new labeling queue.

Options

  • :queue_id - Unique identifier for the queue (required)
  • :schema - LabelSchema defining the label structure (required)
  • :policy - Assignment policy (:round_robin, :random, :expertise) (default: :round_robin)
  • :labels_per_sample - Number of labels needed per sample (default: 1)
  • :assignment_timeout - Timeout in seconds (default: 3600)

export(queue, opts)

@spec export(
  pid() | atom(),
  keyword()
) :: :ok | {:error, term()}

Exports labeled data.

get_next_assignment(queue, labeler_id)

@spec get_next_assignment(pid() | atom(), String.t()) ::
  {:ok, Anvil.Assignment.t()} | {:error, term()}

Gets the next assignment for a labeler.

skip_assignment(queue, assignment_id, opts \\ [])

@spec skip_assignment(pid() | atom(), String.t(), keyword()) ::
  {:ok, Anvil.Assignment.t()} | {:error, term()}

Skips an assignment.

submit_label(queue, assignment_id, values)

@spec submit_label(pid() | atom(), String.t(), map()) ::
  {:ok, Anvil.Label.t()} | {:error, term()}

Submits a label for an assignment.