Producing Messages with Kafka Producers

Kafka Producer : Required properties

Properties props = new Properties();
props.put("bootstrap.servers", "Broker-1:9092, Broker-2:9093"); //Cluster membership: partition leader, etc.
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");//used for message serialization and deserialization
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer")

The properties are used to instantiate an instance of the ProducerConfig class and from there, all producer configuration is defined and referenced internally.

It’s from this object that the internal fields for key and value.serializer are initialized.

Kafka Producer : Optional properties

ProducerRecord(String topic, Integer partition, Long timestamp, K key, V value);
//partition : specific partition within the topic to send ProducerRecord
//timestamp : the Unix timestamp applied to the record
//key : value to be used as basis of determining the partitioning strategy to be employed by kafka producer

Timestamp is controversial because it’s transmitted with the message. It’s a LONG data type, it carries with it the additional overhead of 8 bytes which can affect performance and throughput in high-volume situations.

Kafka Producer : Send messages

KafkaProducer myProducer = new KafkaProducer(props);
ProducerRecord myRecord = new ProducerRecord("my_topic_a", "Key-01", "Value-02"); //Key, Value
myProducer.send(myRecord);//try..catch as best practices

Kafka Producer : Sending message through partitioning strategy

the producer add the producer record object to the specific partition buffer for the topic where it will await the actual send to the broker leader of that partition.

As key is optional, message will be routed using a round robin strategy that attempts to distribute the message across all the partitions in the topic.

If there is a key, need to find if a custom non-default partitioner class was provided as part of configuration properties provided to instantiate the Kafka producer. The producer references to producer config object. If there is no custom non-default partitioner class, message will routed through a key-based partitioning.

Some use cases may call for a custom key-based partitioning scheme. So, you want to develop your own partition (partitioner.class property setting).

Kafka producer : Sending messages with the micro-batching ability

When a producer record has been assigned to a partition through the partitioner, it will get handed over to a RecordAccumulator, where it will be added to a collection of record batch objects for each topic partition combination needed by the producer instance. Finally, record object will be sent off to the brokers.

Kafka producer : Sending messages which are buffered

Each record batch has a limit of how many ProducerRecords can be buffered. the limit is not based on the number of records, but by a configuration setting named batch.size. Across all buffers, there is a configuration setting (buffer.memory) for how much memory can be used to buffer records waiting to be sent to the brokers.