Skip to main content
Question

Generating blank output files

  • September 12, 2023
  • 3 replies
  • 44 views

Anmuthu
Fan

I have a Kafka topic as a source and the sink as a delimited files. I want to generate a blank output file if there is no data in the topic. What is the best way to achieve this. 
 

thanks

Bikram
Headliner
Forum|alt.badge.img+1
  • Headliner
  • September 12, 2023

@Anmuthu

Kindly check if the topic is having record in it or not.

If record != null , then create a file having data in it.

If record is null then configure below code in jython and save the file as txt file.

 

 


Anmuthu
Fan
  • Fan
  • September 12, 2023

Thanks Bikram for the quick reply. I will give that a try. I have another scenario. What happens if the Kafka topic goes through a complex ingestion and generate more than 100 tables where few tables would be empty. What would be the best option in this scenario to generate empty files. 

 

 


Bikram
Headliner
Forum|alt.badge.img+1
  • Headliner
  • September 13, 2023

@Anmuthu 

In that case you can create a metadata table which will contains the table names in it.

By using the JDBC consumer , read data from meta data table which contains only table names in it .

 

for example , the metadata table having tables names in it. Once tables name fetched from the metadata table , then use jdbc lookup processor to check number of records in the table.

 

Attached the pipeline for your reference .

 


Reply