WeaviateEx.Batch.FixedSize (WeaviateEx v0.7.4)

View Source

Fixed-size batch processor for Weaviate batch operations.

Collects objects into a buffer and splits them into fixed-size batches for sending to Weaviate.

Examples

# Create a batcher with batch size of 100
batcher = FixedSize.new(batch_size: 100)

# Add objects
batcher =
  batcher
  |> FixedSize.add_object("Article", %{title: "Article 1"})
  |> FixedSize.add_object("Article", %{title: "Article 2"})

# Get batches for sending
batches = FixedSize.get_batches(batcher)

# Send each batch
Enum.each(batches, fn batch ->
  WeaviateEx.Batch.create_objects(client, batch)
end)

Summary

Types

batch_object()
t()

Functions

add_object(batcher, collection, properties, opts \\ [])

Add an object to the batch buffer.

add_reference(batcher, collection, from_uuid, property, to_target, opts \\ [])

Add a reference to the batch buffer.

buffer_size(batcher)

Get the number of objects currently in the buffer.

clear(batcher)

Clear the object buffer.

get_batches(batcher)

Get batches of objects from the buffer.

get_reference_batches(batcher)

Get batches of references from the buffer.

new(opts \\ [])

Create a new fixed-size batcher.

ready_to_send?(batcher)

Check if the buffer has reached the batch size threshold.

reference_buffer_size(batcher)

Get the number of references currently in the buffer.

Types

batch_object()

@type batch_object() :: %{
  collection: String.t(),
  properties: map(),
  uuid: String.t() | nil,
  vector: [float()] | nil,
  tenant: String.t() | nil
}

t()

@type t() :: %WeaviateEx.Batch.FixedSize{
  batch_size: pos_integer(),
  concurrent_requests: pos_integer(),
  objects_buffer: [batch_object()],
  references_buffer: [map()]
}

Functions

add_object(batcher, collection, properties, opts \\ [])

@spec add_object(t(), String.t(), map(), keyword()) :: t()

Add an object to the batch buffer.

UUID is auto-generated if not provided.

Options

  • :uuid - Custom UUID for the object (auto-generated if not provided)
  • :vector - Custom vector for the object
  • :tenant - Tenant name for multi-tenant collections

Examples

# Auto-generate UUID
batcher
|> FixedSize.add_object("Article", %{title: "Test"})

# With explicit UUID
batcher
|> FixedSize.add_object("Article", %{title: "Test 2"}, uuid: "custom-uuid")

# Deterministic UUID from value
uuid = WeaviateEx.Types.UUID.from_string("Article", "unique-id")
batcher
|> FixedSize.add_object("Article", %{title: "Test"}, uuid: uuid)

add_reference(batcher, collection, from_uuid, property, to_target, opts \\ [])

@spec add_reference(
  t(),
  String.t(),
  String.t(),
  String.t(),
  String.t() | [map()],
  keyword()
) :: t()

Add a reference to the batch buffer.

Supports both single-target and multi-target references.

Single Target

batcher
|> FixedSize.add_reference("Article", "uuid-1", "hasAuthor", "uuid-2")

Multi-Target References

batcher
|> FixedSize.add_reference("Article", "uuid-1", "relatedTo", [
  %{collection: "Article", uuid: "related-uuid-1"},
  %{collection: "Video", uuid: "video-uuid-1"}
])

Options

  • :tenant - Tenant name for multi-tenant collections

buffer_size(batcher)

@spec buffer_size(t()) :: non_neg_integer()

Get the number of objects currently in the buffer.

clear(batcher)

@spec clear(t()) :: t()

Clear the object buffer.

get_batches(batcher)

@spec get_batches(t()) :: [[batch_object()]]

Get batches of objects from the buffer.

Returns a list of batches, each containing up to batch_size objects. The buffer order is preserved (first added = first in batch).

get_reference_batches(batcher)

@spec get_reference_batches(t()) :: [[map()]]

Get batches of references from the buffer.

new(opts \\ [])

@spec new(keyword()) :: t()

Create a new fixed-size batcher.

Options

  • :batch_size - Number of objects per batch (default: 100)
  • :concurrent_requests - Number of concurrent requests (default: 2)

Examples

FixedSize.new()
FixedSize.new(batch_size: 50, concurrent_requests: 4)

ready_to_send?(batcher)

@spec ready_to_send?(t()) :: boolean()

Check if the buffer has reached the batch size threshold.

reference_buffer_size(batcher)

@spec reference_buffer_size(t()) :: non_neg_integer()

Get the number of references currently in the buffer.