Question

Generating blank output files

  • 12 September 2023
  • 3 replies
  • 32 views
Anmuthu
Rank

I have a Kafka topic as a source and the sink as a delimited files. I want to generate a blank output file if there is no data in the topic. What is the best way to achieve this. 
 

thanks

3 replies

Bikram
Rank
Userlevel 5
Badge +1

@Anmuthu

Kindly check if the topic is having record in it or not.

If record != null , then create a file having data in it.

If record is null then configure below code in jython and save the file as txt file.

 

 

Anmuthu
Rank

Thanks Bikram for the quick reply. I will give that a try. I have another scenario. What happens if the Kafka topic goes through a complex ingestion and generate more than 100 tables where few tables would be empty. What would be the best option in this scenario to generate empty files. 

 

 

Bikram
Rank
Userlevel 5
Badge +1

@Anmuthu 

In that case you can create a metadata table which will contains the table names in it.

By using the JDBC consumer , read data from meta data table which contains only table names in it .

 

for example , the metadata table having tables names in it. Once tables name fetched from the metadata table , then use jdbc lookup processor to check number of records in the table.

 

Attached the pipeline for your reference .

 

1 Attachment

Reply


Terms & Conditions