I need to open few json files in side streamsets datacollector pipeline to do some custom processing. Datacollector doesn't allow multiple origins, so I am looking at Javascript scripting stage to open multiple file objects. Also directory origin is opening and displaying only one .json file, so that's too is not working. Any ideas. Thanks
Solved
How to open and read a json file using Javascript scripting stage
- Fan
- 2 replies
Best answer by Bikram
I hope you have configured the precondition in the pipeline finisher as below. You also can try the search pattern sdc*
${record:eventType() == 'no-more-data'}
+1- StreamSets Employee
- 48 replies
Hi
- How does your directory structure of the origin json files look like?
- Can you also share the directory origin configuration which you have setup?
- Fan
- 2 replies
I have attached screenshots for directory structure and directory stage properties and streamsets output. Please take a look. Locations files has 5 records, workers file has 4 records. The output shows just 5 records and one file read
+1- StreamSets Employee
- 48 replies
+1- Headliner
- 486 replies
- Answer
I hope you have configured the precondition in the pipeline finisher as below. You also can try the search pattern sdc*
${record:eventType() == 'no-more-data'}Reply
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.