Question

Can we write the records into/as a text file in Streamsets?

  • 24 June 2022
  • 4 replies
  • 36 views

I have an usecase where we need to read from the source and write all the records into a text file using streamsets. Is it possible to write all the records as a text file?


4 replies

Userlevel 3
Badge

The Local FS destination lets you do this. See our docs.

Hey Dima,thanks for the answer.I ’d like know if we can write directly as a text file to NAS location(using SFTP) ?

Userlevel 3
Badge

The SFTP destination only writes directly via the Whole File format. Your best bet would be to write to Local FS and then have a separate process move the complete file via SFTP. 

Userlevel 2
Badge +1

@nikkichow 

In this case , you can create the first file to read data from source and send to s3 bucket as txt file format.

In the second pipeline , read file from s3 bucket as whole file and send sftp server.

 

Reply