Module brod_group_subscriber_v2

Description
Data Types
Function Index
Function Details

This module implements an improved version of brod_group_subscriber behavior.

Behaviours: brod_group_member, gen_server.

This module defines the brod_group_subscriber_v2 behaviour.
Required callback functions: init/2, handle_message/2.
Optional callback functions: assign_partitions/3, get_committed_offset/3, terminate/2.

Description

This module implements an improved version of brod_group_subscriber behavior. Key difference is that each partition worker runs in a separate Erlang process, allowing parallel message processing.

Data Types

commit_fun()

commit_fun() = fun((brod:offset()) -> ok)

init_info()

init_info() = #{group_id := brod:group_id(), topic := brod:topic(), partition := brod:partition(), commit_fun := commit_fun()}

member_id()

member_id() = brod:group_member_id()

subscriber_config()

subscriber_config() = #{client := brod:client(), group_id := brod:group_id(), topics := [brod:topic()], cb_module := module(), init_data => term(), message_type => message | message_set, consumer_config => brod:consumer_config(), group_config => brod:group_config()}

Function Index

ack/4	Commit offset for a topic-partition, but don't commit it to Kafka.
assign_partitions/3	This function is called only when `partition_assignment_strategy` is set for `callback_implemented` in group config.
assignments_received/4	Called by group coordinator when there is new assignment received.
assignments_revoked/1	Called by group coordinator before re-joining the consumer group.
commit/4	Ack offset for a topic-partition.
get_committed_offsets/2	Called by group coordinator when initializing the assignments for subscriber.
handle_call/3
start_link/1	Start (link) a group subscriber.
stop/1	Stop group subscriber, wait for pid `DOWN` before return.

Function Details

ack/4

ack(Pid::pid(), Topic::brod:topic(), Partition::brod:partition(), Offset::brod:offset()) -> ok

Commit offset for a topic-partition, but don't commit it to Kafka. This is an asynchronous call

assign_partitions/3

assign_partitions(Pid::pid(), Members::[brod:group_member()], TopicPartitionList::[brod:topic_partition()]) -> [{member_id(), [brod:partition_assignment()]}]

This function is called only when partition_assignment_strategy is set for callback_implemented in group config.

assignments_received/4

assignments_received(Pid::pid(), MemberId::member_id(), GenerationId::integer(), TopicAssignments::brod:received_assignments()) -> ok

Called by group coordinator when there is new assignment received.

assignments_revoked/1

assignments_revoked(Pid::pid()) -> ok

Called by group coordinator before re-joining the consumer group.

commit/4

commit(Pid::pid(), Topic::brod:topic(), Partition::brod:partition(), Offset::brod:offset()) -> ok

Ack offset for a topic-partition. This is an asynchronous call

get_committed_offsets/2

get_committed_offsets(Pid::pid(), TopicPartitions::[brod:topic_partition()]) -> {ok, [{brod:topic_partition(), brod:offset()}]}

Called by group coordinator when initializing the assignments for subscriber.

NOTE: This function is called only when offset_commit_policy is set to consumer_managed in group config.

NOTE: The committed offsets should be the offsets for successfully processed (acknowledged) messages, not the begin_offset to start fetching from.

handle_call/3

handle_call(Call, From, State) -> any()

start_link/1

start_link(Config::subscriber_config()) -> {ok, pid()} | {error, any()}

Start (link) a group subscriber.

Possible Config keys:

client: Client ID (or pid, but not recommended) of the brod client. Mandatory
group_id: Consumer group ID which should be unique per kafka cluster. Mandatory
topics: Predefined set of topic names to join the group. Mandatory
NOTE: The group leader member will collect topics from all members and assign all collected topic-partitions to members in the group. i.e. members can join with arbitrary set of topics.
cb_module: Callback module which should have the callback functions implemented for message processing. Mandatory
group_config: For group coordinator, see brod_group_coordinator:start_link/6 Optional
consumer_config: For partition consumer, brod_topic_subscriber:start_link/6. Optional
message_type: The type of message that is going to be handled by the callback module. Can be either message or message set. Optional, defaults to message
init_data: The term() that is going to be passed to CbModule:init/2 when initializing the subscriber. Optional, defaults to undefined

stop/1

stop(Pid::pid()) -> ok

Stop group subscriber, wait for pid DOWN before return.

Generated by EDoc