Kaffe.GroupMember (Kaffe v1.25.0) View Source

Note: The brod_group_member behavior is used.

Consume messages from a Kafka topic for a consumer group. There is one brod group member per topic! So as new topics are added to configuration so are the number of brod group members. Likewise, if you're using something like Heroku Kafka and have multiple dynos for your consumer, there will be a Kaffe.GroupMember per dyno and each group member will receive a equal set of partition assignments for each topic.

The actual consumption is delegated to a series of subscribers, see Kaffe.Subscriber for details on how messages are processed.

The subscribers are assigned generations. Each generation represents a specific configuration. In order to allow the partitions to be rebalanced on startup, there is a delay between receiving a set of assignments associated with that generation and actually creating the subscribers. If a new generation is received in the mean time, the older generation is discarded.

See the follow for more details: https://github.com/klarna/brod/blob/master/src/brod_group_member.erl https://github.com/klarna/brucke/blob/master/src/brucke_member.erl

Link to this section Summary

Functions

Callback implementation for c::brod_group_member.assign_partitions/3.

Callback implementation for c::brod_group_member.assignments_received/4.

Callback implementation for c::brod_group_member.assignments_revoked/1.

Returns a specification to start this module under a supervisor.

Callback implementation for c::brod_group_member.get_committed_offsets/2.

Callback implementation for GenServer.init/1.

Link to this section Functions

Link to this function

assign_partitions(pid, members, topic_partitions)

View Source

Callback implementation for c::brod_group_member.assign_partitions/3.

Link to this function

assignments_received(pid, member_id, generation_id, assignments)

View Source

Callback implementation for c::brod_group_member.assignments_received/4.

Link to this function

assignments_revoked(pid)

View Source

Callback implementation for c::brod_group_member.assignments_revoked/1.

Returns a specification to start this module under a supervisor.

See Supervisor.

Link to this function

get_committed_offsets(group_member_pid, topic_partitions)

View Source

Callback implementation for c::brod_group_member.get_committed_offsets/2.

Callback implementation for GenServer.init/1.

Link to this function

start_link(subscriber_name, consumer_group, worker_manager_pid, topic)

View Source