/
Managing Data Streams

Managing Data Streams

Owned by thierry sinassamy
Last updated: Mar 08, 2022
2 min read

Data Stream = Kafka Topic + Data Governance

Admin Client and Data Governance

Admin Client
Data Governance

Problems related to Schemas, Lineage,… cannot be solved by the Admin Client itself.

How to set up Admin Client

  1. Define Properties : bootstrap.servers=broker-1:8684

  2. Create Admin Client : adminClient = new AdminClient<T>(properties)

  3. Perform Action such as create topic = adminClient.createTopic(topic)

Deleting Records by Streaming Application

This action is more complex because specific records in topic have been incorrectly produced. Kafka does not store data like a database. So, we can’t delete individual records. We need to implement a filter pattern.

Records filtered by a streaming application created by KS/KSQL

Deleting Records by Offset

We can remove all records before a certain offset. With a single partition, it’s pretty simple, but with several partitions, it could become more complex.

If several partitions, we need to specify the partition as well.

Removing before Offset 3 by Admin Client

Deleting Records by Key

Removing by key by Kafka Producer and Apache Kafka