file transfer from origin to destination

@Santhosh Kumar 

In this case you can put all the files in s3 bucket and configure multiple destination where you need to send the data .

This can be possible in StreamSets data collector.

If you have the files from multiple source then you can use StreamSets transformer and send it multiple destination.

If you are having any issues , please let me know .

Thanks & Regards

Bikram_

multiple file from one local directories to transfer another local directories is possible . suggest any ideo to achieve this

@Santhosh Kumar 

That can be possible but we need to filter out which file needs to transfer in which directory if its needed..

From origin , fetch all the files and add filter using StreamSets collector  to send files to different directory.

May I know if you are planning to send all the files to different directories. If that is case then no need to add filter in between them.

If you want to send some specific files to specific directory then add filter in between origin and destination.

I don’t have any video for that  but I can provide pipeline for the same.

Thanks & Regards

Bikram_

can you please provide the pipeline and details .

@Santhosh Kumar 

I will create pipeline and share with you . May I know if its fine to you if i share the pipeline by tomorrow .

@Santhosh Kumar 

Please find attached the complete pipeline.

Steps 1 : Origin directory contains 2 .csv files in it. and destination path contains 2 directory named as zomato1 and zomato2

Step2 : Read the files from origin as whole file.

step3 : Added filter , if the filename is origin_taxi.csv then it will deliver  to zomato1 directory.

step4 : Else it will deliver to zomato2 directory .

Kindly import the directory in your CDC and change the directory path and keep the files which you want to send to different destination folders based on your need and check if it helps.

Please let me know if you are having any issues.

Thanks & Regards

Bikram_

@Bikram 

my use case is multiple files in directory ,these files are transferred to another directory as whole file format .but during processing i got error like filename is already exists.

@Santhosh Kumar 

I did some changes in the  the pipeline  and i hope it will solve your issue.

Kindly let me know if you have any issues.

Thanks & Regards

Bikram_