I have a Kafka topic as a source and the sink as a delimited files. I want to generate a blank output file if there is no data in the topic. What is the best way to achieve this.
thanks
I have a Kafka topic as a source and the sink as a delimited files. I want to generate a blank output file if there is no data in the topic. What is the best way to achieve this.
thanks
In that case you can create a metadata table which will contains the table names in it.
By using the JDBC consumer , read data from meta data table which contains only table names in it .
for example , the metadata table having tables names in it. Once tables name fetched from the metadata table , then use jdbc lookup processor to check number of records in the table.
Attached the pipeline for your reference .
Thanks Bikram for the quick reply. I will give that a try. I have another scenario. What happens if the Kafka topic goes through a complex ingestion and generate more than 100 tables where few tables would be empty. What would be the best option in this scenario to generate empty files.
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.