I have an usecase where we need to read from the source and write all the records into a text file using streamsets. Is it possible to write all the records as a text file?
In this case , you can create the first file to read data from source and send to s3 bucket as txt file format.
In the second pipeline , read file from s3 bucket as whole file and send sftp server.
The SFTP destination only writes directly via the Whole File format. Your best bet would be to write to Local FS and then have a separate process move the complete file via SFTP.
Hey Dima,thanks for the answer.I ’d like know if we can write directly as a text file to NAS location(using SFTP) ?
The Local FS destination lets you do this. See our docs.
Reply
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.