I have a Kafka topic as a source and the sink as a delimited files. I want to generate a blank output file if there is no data in the topic. What is the best way to achieve this.
thanks
I have a Kafka topic as a source and the sink as a delimited files. I want to generate a blank output file if there is no data in the topic. What is the best way to achieve this.
thanks
Kindly check if the topic is having record in it or not.
If record != null , then create a file having data in it.
If record is null then configure below code in jython and save the file as txt file.
Thanks Bikram for the quick reply. I will give that a try. I have another scenario. What happens if the Kafka topic goes through a complex ingestion and generate more than 100 tables where few tables would be empty. What would be the best option in this scenario to generate empty files.
In that case you can create a metadata table which will contains the table names in it.
By using the JDBC consumer , read data from meta data table which contains only table names in it .
for example , the metadata table having tables names in it. Once tables name fetched from the metadata table , then use jdbc lookup processor to check number of records in the table.
Attached the pipeline for your reference .
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.