How to build a reliable producer / consumer with Kafka

ProducerConfig Object has to be instantiated with the proper settings to build a reliable producer.

var _producerConfig = new ProducerConfig
{
        BootstrapServers = bootstrapServer, // initiates list of brokers
        EnableDeliveryReports = true, //enabled notifications to delivery reports
        ClientId = Dns.GetHostName(), // client identifier
        // retry settings:       
        Acks = Acks.All,  // Receive acknowledgement from all sync replicas
        MessageSendMaxRetries = 3, // Number of times to retry before giving up      
        RetryBackoffMs = 1000,  // Duration to retry before next attempt
        // Set to true if you don't want to reorder messages on retry
        EnableIdempotence = true
};

ConsumerConfig Object has to be instantiated with the proper settings to build a reliable consumer.

var _consumerConfig = new ConsumerConfig
{
      BootstrapServers = bootstrapServer,// initiates list of brokers
      EnableAutoCommit = false, // No Automatically and periodically commit offsets in the background.
      EnableAutoOffsetStore = false,//No Automatically store offset of last message provided to application
      MaxPollIntervalMs = 300000,//Maximum allowed time between calls to consume messages
      GroupId = "default",

      // Read messages from start if no commit exists.
      AutoOffsetReset = AutoOffsetReset.Earliest
};

Handling Streaming Data with Messaging Systems ( Kafka & Azure Event Hub)

How to build a reliable producer / consumer with Kafka

Analytics