Managing Data Streams

Owned by thierry sinassamy
Last updated: Mar 08, 2022
2 min read

Data Stream = Kafka Topic + Data Governance

Admin Client and Data Governance

Admin Client

Problems related to Schemas, Lineage,… cannot be solved by the Admin Client itself.

How to set up Admin Client

  1. Define Properties : bootstrap.servers=broker-1:8684

  2. Create Admin Client : adminClient = new AdminClient<T>(properties)

  3. Perform Action such as create topic = adminClient.createTopic(topic)

Deleting Records by Streaming Application

This action is more complex because specific records in topic have been incorrectly produced. Kafka does not store data like a database. So, we can’t delete individual records. We need to implement a filter pattern.

Deleting Records by Offset

We can remove all records before a certain offset. With a single partition, it’s pretty simple, but with several partitions, it could become more complex.

Deleting Records by Key

Â