StreamSets pipelines set up with a Directory origin only query directories when they're running. The way folks would generally do this would be to have a pipeline running continuously and, as files are added, that pipeline will pick up and process any new files that appear. 

@Rohith 

Everything @dima said,

plus: 

You can chose to read data in your directory in alphabetical order or as per datetime of creation. StreamSets uses OFFSETs to keep track of already processed files. So choose your sorting method as per your requirement. If your files always have alphabetically incremental names then use first option else use last change datetime as your sorting method.