Skip to main content

Hi Guys,

 

I have a requirement where I have thousands of json file url’s in a file(all files are of same format). I need to process the data for every file and load data in to destination.

 

Ex: local_dir/file_list.txt

https://example1.json

https://example2.json

https://example3.json

https://example4.json

Hello @va7777 - Thank you for reaching out to the StreamSets Community. You can use the directory origin and pass the *.json as a file name pattern so that it can pick up all the files with .json format with the  1000 record per batch. 

What is going to be destination in this case?

 

@va7777 

You can keep the *.txt file in local directory and fetch the file data . Then pass file entry to http client to fetch data from above mentioned http URLs.

If you are having issues , please let me know .

Please construct your pipeline as below .

 

 

Thanks & Regards

Bikram_

Thank you @Bikram @AkshayJadhav for your answers. It worked.

Reply


Terms & Conditions