Data Stream = Kafka Topic + Data Governance
Admin Client and Data Governance
Problems related to Schemas, Lineage,… cannot be solved by the Admin Client itself.
How to set up Admin Client
Define Properties : bootstrap.servers=broker-1:8684
Create Admin Client : adminClient = new AdminClient<T>(properties)
Perform Action such as create topic = adminClient.createTopic(topic)
Deleting Records by Streaming Application
This action is more complex because specific records in topic have been incorrectly produced. Kafka does not store data like a database. So, we can’t delete individual records. We need to implement a filter pattern.
Deleting Records by Offset
We can remove all records before a certain offset. With a single partition, it’s pretty simple, but with several partitions, it could become more complex.
If several partitions, we need to specify the partition as well.
Deleting Records by Key