How to handle countless partitions for a consumer perspective ? Having a bunch of consumers independently consuming messages from topics and partitions won’t solve this challenge of scalability.
Consumers have to work together ! So what ?
Consumer Groups is the solution. IT’s a collection of individual independent consumers processes working together as a team.
Joining a group is configurable with the setting “group.id ”. The consumer group enables the sharing of the messages consumption and of the processing load : high level of throughput and parallelism; upgrade of performance; increase of levels of redundancy (failure of a single consumer).
How to rebalance a consumer group if needed ?