Solved

How to open and read a json file using Javascript scripting stage

  • 14 June 2022
  • 5 replies
  • 137 views

I need to open few json files in side streamsets datacollector pipeline to do some custom processing. Datacollector doesn't allow multiple origins, so I am looking at Javascript scripting stage to open multiple file objects.  Also directory origin is opening and displaying only one .json file, so that's too is not working. Any ideas. Thanks

icon

Best answer by Bikram 19 June 2022, 15:20

View original

5 replies

Userlevel 3
Badge +1

Hi @rajneesh75 ,

  1. How does your directory structure of the origin json files look like? 
  2. Can you also share the directory origin configuration which you have setup?

I have attached screenshots for directory structure and directory stage properties and streamsets output. Please take a look. Locations files has 5 records, workers file has 4 records. The output shows just 5 records and one file read

 

Userlevel 3
Badge +1

@rajneesh75 What is the File Name Pattern Mode and Read Order are set to? Also what is the Event that is set on Directory origin to be consumed?

screenshot attached

Userlevel 5
Badge +1

 

@rajneesh75 

I hope you have configured the  precondition in the pipeline finisher as below. You also can try the search pattern sdc* 

 

${record:eventType() == 'no-more-data'}

Reply