Skip to main content
Question

how to create the pipeline for the xml data where it is taken from sftp

  • May 2, 2024
  • 2 replies
  • 42 views

moviepages
Fan

how to create the pipeline for the xml data where it is taken from sftp in the streamsets we have getting only 1 record from sftp in the preview data 

2 replies

Bikram
Headliner
Forum|alt.badge.img+1
  • Headliner
  • 486 replies
  • May 2, 2024

@moviepages 

 

 

While it may display only one record in the preview, please execute the pipeline to verify if it successfully retrieved all the data from the source file.


roma
Fan
  • Fan
  • 4 replies
  • May 9, 2024

SFTP origin delivers data in Whole File format, that’s why you see one record -- it’s only a record with the reference to the file.

Your destination should also be in Whole File format. Then you can (if you wish) read inside the file using a second pipeline.