I want my pipeline to automatically trigger only when the source file is placed on the server. How can we achieve this in streamsets?
- StreamSets Employee
StreamSets pipelines set up with a Directory origin only query directories when they're running. The way folks would generally do this would be to have a pipeline running continuously and, as files are added, that pipeline will pick up and process any new files that appear.
- Headliner
Everything
plus:
You can chose to read data in your directory in alphabetical order or as per datetime of creation. StreamSets uses OFFSETs to keep track of already processed files. So choose your sorting method as per your requirement. If your files always have alphabetically incremental names then use first option else use last change datetime as your sorting method.
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.