Topics
- Kafka topics are really just a named feed or category of messages.
- Kafka topics are a logical entity.
- for each topic, Kafka cluster maintains one or more physical log files.
Each message represents an event or fact that from the perspective of the producer, it will be available to potential consumers.
The message are immutable. Once they are received into a topic, they cannot be changed. So, if a message is not valid, it will be not valid in the topic. And , the consumer has to reconcile between the messages when it reads and processes them.
Note : Message 3 (in red) is invalid. So, the only recourse is to follow up that invalid message with a new message (Message 5 in green).
Each message has a Timestamp, a Referenceable identifier, a binary payload of data.
How do the consumers maintain their autonomy as far as message consumption from a common topic ? IT’s called the message offset. IT’s how consumers can do read messages at their own pace.
OFFSET = PLACEHOLDER AS THE LAST READ MESSAGE POSITION + MAINTAINED BY THE KAFKA CONSUMER + REFERS TO A MESSAGE IDENTIFIER
Partitions
Brokers
...