| Note |
|---|
How to handle countless partitions for a consumer perspective ? Having a bunch of consumers independently consuming messages from topics and partitions won’t solve this challenge of scalability. |
Consumers have to work together ! So what ?
Consumer Groups is the solution. IT’s a collection of individual independent consumers processes working together as a team.
...
How to rebalance a consumer group if needed ?
When a new consumer is assigned to a partition, it needs to know what offset it should start from because it does not a current position for this particular partition. Fortunately, the last committed offset has been cached from the previous consumer by the consumers subscription state object. From now, the consumers subscription state object can instruct the new consumer that it will start with offset 4.
...
Rebalancing protocol is also initiated during a consumer failure, when topic change (partition added).