Kafka Producer Batching : https://www.conduktor.io/kafka/kafka-producer-batching
A producer will have up to 5 requests in flight (controlled by the max.in.flight.requests.per.connection setting), meaning up to 5 message batches will be sent at the same time.
This smart batching allows Kafka to increase throughput while maintaining very low latency. Batches have a higher compression ratio so a better disk and networking efficiency. Batching is mainly controlled by two producer settings - linger.ms and batch.size.
Producer Batching and no message compression
By default, the messages are not compressed.
...
| Info |
|---|
The max size is represented by the setting “batch.size”. The default value of batch.size is 16 KB so,16Kb x 1024 bytes = 16384 bytes (binary and not decimal). We can increase to 32KB or 64KB, so it can help the increase the compression, throughput and efficiency of requests. |
| Code Block | ||
|---|---|---|
| ||
var producerConfig = new ProducerConfig
{
Acks = Acks.All,
BootstrapServers = "localhost:9092",
BatchSize = 32768, //32KB more than the default
LingerMs = 20 //some lag or delay to increase the throughput and compression
}; |
Producer Batching with message compression enabled
Compression is enabled on the producer-side event if we can do it on the broker-side.
...
| Code Block |
|---|
var producerConfig = new ProducerConfig { Acks = Acks.All, BootstrapServers = "localhost:9092", BatchSize = 32768, LingerMs = 20, CompressionType = CompressionType.Snappy // or lz4 }; |
| Note |
|---|
Speed versus Compression : Consider testing snappy or lz4 for optimal speed / compression ratio. |