I am trying to create a pipeline which gets triggered when I add a file to a directory. Is there any functionality in the streamsets hub or any kind of trigger which I can directly set to the “Directory” origin which can in turn trigger the pipeline ahead?
If you use Directory origin and run your pipeline with relevant options to read alphabetically or time based, then a new file in that directory will generate an event that can be used to do what you want to achieve. Also, you can use Orchestration libraries to trigger another job based on the event - new-file
If you use Directory origin and run your pipeline with relevant options to read alphabetically or time based, then a new file in that directory will generate an event that can be used to do what you want to achieve. Also, you can use Orchestration libraries to trigger another job based on the event - new-file
Thanks for the response. Can you please brief me more on the orchestration library? I am not able to use it in the way you have described.
Reply
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.