This depends on ZooKeeper to rebalancing and managing consumer group

zookeeper structure

It will build the structure on zookeeper like below after consumer started:

  /cafex
   |-- topic
   |  |-- group_name
   |  |  |-- leader
   |  |  |-- consumers
   |  |  |  |-- balance
   |  |  |  |  |-- cafex@192.168.0.1       - [0,1,2,3]     # persistent
   |  |  |  |  |-- cafex@192.168.0.2       - [4,5,6,7]     # persistent
   |  |  |  |  |-- cafex@192.168.0.3       - [8,9,10,11]   # persistent
   |  |  |  |-- online
   |  |  |  |  |-- cafex@192.168.0.1                       # ephemeral
   |  |  |  |  |-- cafex@192.168.0.2                       # ephemeral
   |  |  |  |-- offline
   |  |  |  |  |-- cafex@192.168.0.3                       # persistent Deprecated
   |  |  |-- locks

First of all, every consumer will register itself under the node consumers/online. All consumer will elected a leader, which will be responsible for balancing. Leader collects all sub node under the consumers/online and consumers/offline, and executes the balancing, writes the result to every consumer node under the consumers/balance node. All consumers is listening on the corresponding node under the consumers/balance for changes to adjust the partition workers.

NOTE: For now, the consumer will use the erlang node name as its name to register on the zookeeper, so make sure starting with the -name argument.