Skip to main content

I need to open few json files in side streamsets datacollector pipeline to do some custom processing. Datacollector doesn't allow multiple origins, so I am looking at Javascript scripting stage to open multiple file objects.  Also directory origin is opening and displaying only one .json file, so that's too is not working. Any ideas. Thanks

Hi @rajneesh75 ,

  1. How does your directory structure of the origin json files look like? 
  2. Can you also share the directory origin configuration which you have setup?

I have attached screenshots for directory structure and directory stage properties and streamsets output. Please take a look. Locations files has 5 records, workers file has 4 records. The output shows just 5 records and one file read

 

@rajneesh75 What is the File Name Pattern Mode and Read Order are set to? Also what is the Event that is set on Directory origin to be consumed?

screenshot attached

 

@rajneesh75 

I hope you have configured the  precondition in the pipeline finisher as below. You also can try the search pattern sdc* 

 

${record:eventType() == 'no-more-data'}

Reply


Terms & Conditions