Question

File watcher event in streamsets

  • 11 November 2022
  • 2 replies
  • 42 views

I want my pipeline to automatically trigger only when the source file is placed on the server. How can we achieve this in streamsets?


2 replies

Userlevel 3
Badge

StreamSets pipelines set up with a Directory origin only query directories when they're running. The way folks would generally do this would be to have a pipeline running continuously and, as files are added, that pipeline will pick up and process any new files that appear. 

Userlevel 3
Badge

@Rohith 

Everything @dima said,

plus: 

You can chose to read data in your directory in alphabetical order or as per datetime of creation. StreamSets uses OFFSETs to keep track of already processed files. So choose your sorting method as per your requirement. If your files always have alphabetically incremental names then use first option else use last change datetime as your sorting method.

Reply