View Source KafkaEx.Producer.Partitioner behaviour (kafka_ex v1.0.0-rc.1)

Behaviour definition for partitioners that assign partitions for produce requests.

A partitioner is responsible for determining which partition a message should be sent to when no explicit partition is provided.

Implementing a Custom Partitioner

To implement a custom partitioner, create a module that implements the assign_partition/4 callback:

defmodule MyApp.RoundRobinPartitioner do
  @behaviour KafkaEx.Producer.Partitioner

  def assign_partition(_topic, _key, _value, partition_count) do
    # Simple round-robin - in practice you'd want state for this
    :rand.uniform(partition_count) - 1
  end
end

Then configure it in your application:

config :kafka_ex, partitioner: MyApp.RoundRobinPartitioner

Built-in Partitioners

Summary

Callbacks

Assigns a partition for the given message.

Functions

Returns the configured partitioner module.

Callbacks

Link to this callback

assign_partition(topic, key, value, partition_count)

View Source
@callback assign_partition(
  topic :: String.t(),
  key :: binary() | nil,
  value :: binary(),
  partition_count :: pos_integer()
) :: non_neg_integer()

Assigns a partition for the given message.

Parameters

  • topic - The topic name
  • key - The message key (may be nil)
  • value - The message value
  • partition_count - The number of partitions for the topic

Returns

The partition number (0-indexed) to send the message to.

Functions

@spec get_partitioner() :: module()

Returns the configured partitioner module.

Defaults to KafkaEx.Producer.Partitioner.Default if not configured.