Skip to main content

Hello Experts,

 

I am new to the Streamsets. I am doing POC on SFTP origin component. My requirement is moving or copying the files between directories in SFTP and valid file should be copied into S3.

we are having the following directory structure.

  1. Parent Directory name (By department) and it has the following sub directories
    1. Ready
    2. InProgress
    3. Reject
    4. Finish

Step1: Users will upload the file into Ready folder. File processing pipe line should check whether do we have file in the ready folder. If it is available move or copy the file into “InProgess” otherwise finish the pipeline

Step2: If file available in ready, it should move or copy to “INProgress”, during the copying the files into S3 in case of any errors files should be moved to Reject directory.

Step3: If no errors file should copy into S3 and move the file from InProgress to Finish directory and finish the pipe file.

@kishorerct 

please find attached pipeline and it will help you to handle your case.

 

Pipeline 1:

This is will be read file from your “Ready” directory   and send to in-progress directory if file exists in it.

 

Pipeline 2:

This will check the progress directory and also check if file exists  in it or not , if file exists then , you can use jython processor to validate the file if it’s readable or not . If there are any error then it will stop the pipeline.

 

Create two jobs for pipeline1 and pipeline2.

Create an orchestration pipeline for the jobs created .

 

I hope this will help you on your case.

 

Please let me know if you are facing any issues , i am happy to help you.

 

Thanks & Regards,

Bikram_ 

 

https://docs.streamsets.com/platform-datacollector/latest/datacollector/UserGuide/Orchestration_Pipelines/OrchestrationPipelines_Title.html

Reply


Terms & Conditions