Question:
How to control the interval at which an SFTP directory should be checked or polled?
Answer:
The SFTP connector from SDC polls the SFTP server for new files only when the connector's queue is empty. The polling frequency is defined by the Data Collector when it asks for more records to the origin. This is a non-configurable entity (as of this writing). Base on how Data Collector gathers records, there exists a property in the UI ("Batch Wait Time (ms)"), that can be used to setup how often Data Collector requests records to the origin. Varying this value should help to control this.
For example, "Batch Wait Time (ms)" is the time the pipeline waits before sending across a complete/partial batch, it has queued locally, to the destination. That's the time after which the origin would poll for new files/records.
So, if you were to have a file/set-of-files at 't = T1(sec)' then they'd all get queued in the batch at polling interval 'P1(ms)'. If you have new files coming in at 't = T1+a(sec)', then that'd be picked in the next wave spaced 'P1+"Batch Wait Time (ms)"'.