I am trying to create a pipeline which gets triggered when I add a file to a directory. Is there any functionality in the streamsets hub or any kind of trigger which I can directly set to the “Directory” origin which can in turn trigger the pipeline ahead?
Page 1 / 1
If you use Directory origin and run your pipeline with relevant options to read alphabetically or time based, then a new file in that directory will generate an event that can be used to do what you want to achieve. Also, you can use Orchestration libraries to trigger another job based on the event - new-file
If you use Directory origin and run your pipeline with relevant options to read alphabetically or time based, then a new file in that directory will generate an event that can be used to do what you want to achieve. Also, you can use Orchestration libraries to trigger another job based on the event - new-file
Thanks for the response. Can you please brief me more on the orchestration library? I am not able to use it in the way you have described.
If you are using DataOps Platform then enable Orchestration library in your deployment. This will enable processors like Start Job
If you are using SCH 3.x then for your data collector install Orchestration library in the Package Manager
Reply
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.