Skip to main content
Question

Can we write the records into/as a text file in Streamsets?


nikkichow
Fan

I have an usecase where we need to read from the source and write all the records into a text file using streamsets. Is it possible to write all the records as a text file?

dima
StreamSets Employee
Forum|alt.badge.img
  • StreamSets Employee
  • June 24, 2022

The Local FS destination lets you do this. See our docs.


nikkichow
Fan

Hey Dima,thanks for the answer.I ’d like know if we can write directly as a text file to NAS location(using SFTP) ?


dima
StreamSets Employee
Forum|alt.badge.img
  • StreamSets Employee
  • June 24, 2022

The SFTP destination only writes directly via the Whole File format. Your best bet would be to write to Local FS and then have a separate process move the complete file via SFTP. 


Bikram
Headliner
Forum|alt.badge.img+1
  • Headliner
  • June 26, 2022

@nikkichow 

In this case , you can create the first file to read data from source and send to s3 bucket as txt file format.

In the second pipeline , read file from s3 bucket as whole file and send sftp server.

 


Reply