Solved

File list-url's in a file to process


Hi Guys,

 

I have a requirement where I have thousands of json file url’s in a file(all files are of same format). I need to process the data for every file and load data in to destination.

 

Ex: local_dir/file_list.txt

https://example1.json

https://example2.json

https://example3.json

https://example4.json

icon

Best answer by Bikram 2 June 2023, 08:32

View original

3 replies

Userlevel 4
Badge

Hello @va7777 - Thank you for reaching out to the StreamSets Community. You can use the directory origin and pass the *.json as a file name pattern so that it can pick up all the files with .json format with the  1000 record per batch. 

What is going to be destination in this case?

 

Userlevel 5
Badge +1

@va7777 

You can keep the *.txt file in local directory and fetch the file data . Then pass file entry to http client to fetch data from above mentioned http URLs.

If you are having issues , please let me know .

Please construct your pipeline as below .

 

 

Thanks & Regards

Bikram_

Thank you @Bikram @AkshayJadhav for your answers. It worked.

Reply