Solved

Need Help in Reading data from SFTP location and loading the same data Another system VIA REST call

  • 16 June 2022
  • 4 replies
  • 147 views

Need Help in Reading data from SFTP location and loading the same data into another system VIA REST call (HTTP)

 

 

Does someone help me with sample pipeline?

 

icon

Best answer by Rishi 17 June 2022, 13:02

View original

4 replies

Userlevel 4
Badge

@MohammedSohail You can simply create the `SFTP → HTTP Client Destination` pipeline to achieve your use-case. 

 

Please use the Stage documentation for more details.

SFTP : here

HTTP Client Destination: here

@MohammedSohail You can simply create the `SFTP → HTTP Client Destination` pipeline to achieve your use-case. 

 

Please use the Stage documentation for more details.

SFTP : here

HTTP Client Destination: here

Thanks @Rishi  I want to send a file as attachment over HTTP call (REst api), is it possible? Do you have a sample pipeline where we are sending a file over http rest call?

@MohammedSohail You can simply create the `SFTP → HTTP Client Destination` pipeline to achieve your use-case. 

 

Please use the Stage documentation for more details.

SFTP : here

HTTP Client Destination: here

@Rishi Can you suggest here. I am not sure how to send file as attachment via Rest call

Userlevel 4
Badge

Hi @MohammedSohail  One of the possible approach could be as below. 

SFTP Origin : This will read file from SFTP server.

LocalFS: We will use LoacalFS destination for staging this files

 

Note: use the whole file format as Data Format in both the stage. This way will get the exact file to localfs.

https://docs.streamsets.com/portal/datacollector/latest/help/datacollector/UserGuide/Data_Formats/WholeFile.html#concept_nfc_qkh_xw 

 

Now configure the event for localfs destination which will trigger shell event.  Which internally will call curl command like below 

$ curl -F ‘data=@path/to/local/file’ UPLOAD_ADDRESS 

you can find more details about curl command here

Reply