(S)FTP origin can’t tail file that is actively written into. This is because the origin can’t pass the file descriptor from server to client.
For reliable tailing the tail have to run locally and get file descriptor from OS. File descriptors can’t be transferred across network, because they are not a name nor a variable, they are a resource inside operating system that is exposed to the process inside.
There are two possible workarounds:
- Running the SDC on the same server as SFTP and create pipeline with File Tail origin to read active tailed files.
- Or (but this is not a real time option) wait till the file is rotated, move it to a directory on the same server as SFTP and use SFTP origin to read the file.