Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 8 Next »

Kafka Producer Batching : https://www.conduktor.io/kafka/kafka-producer-batching

A producer will have up to 5 requests in flight (controlled by the max.in.flight.requests.per.connection setting), meaning up to 5 message batches will be sent at the same time.

This smart batching allows Kafka to increase throughput while maintaining very low latency. Batches have a higher compression ratio so a better disk and networking efficiency. Batching is mainly controlled by two producer settings - linger.ms and batch.size.

Producer Batching and no message compression

The max size is represented by the setting “batch.size”. The default value of batch.size is 16 KB so,16Kb x 1024 bytes = 16384 bytes (binary and not decimal). We can increase to 32KB or 64KB, so it can help the increase the compression, throughput and efficiency of requests.

var producerConfig = new ProducerConfig
{
       Acks = Acks.All,
       BootstrapServers = "localhost:9092",
       BatchSize = 32768, //32KB more than the default
       LingerMs = 20 //some lag or delay to increase the throughput and compression
 };

Producer Batching with message compression

  • No labels