Skip to main content
Question

Configuring a trigger to activate when a file is created in a directory

H
Fan

I am trying to create a pipeline which gets triggered when I add a file to a directory. Is there any functionality in the streamsets hub or any kind of trigger which I can directly set to the “Directory” origin which can in turn trigger the pipeline ahead? 

3 replies

saleempothiwala
Headliner
Forum|alt.badge.img

@harshit_agrawal1 

 

If you use Directory origin and run your pipeline with relevant options to read alphabetically or time based, then a new file in that directory will generate an event that can be used to do what you want to achieve. Also, you can use Orchestration libraries to trigger another job based on the event - new-file

 

 

H
Fan
saleempothiwala wrote:

@harshit_agrawal1

 

If you use Directory origin and run your pipeline with relevant options to read alphabetically or time based, then a new file in that directory will generate an event that can be used to do what you want to achieve. Also, you can use Orchestration libraries to trigger another job based on the event - new-file

 

 

Thanks for the response. Can you please brief me more on the orchestration library? I am not able to use it in the way you have described.

saleempothiwala
Headliner
Forum|alt.badge.img

@harshit_agrawal1 

If you are using DataOps Platform then enable Orchestration library in your deployment. This will enable processors like Start Job

If  you are using SCH 3.x then for your data collector install Orchestration library in the Package Manager

Reply


Terms & Conditions