Solved

how to load data with embedded api calls

  • 5 June 2023
  • 2 replies
  • 17 views
va7777
Rank

Can you please help with how I can address this problem. I am using SDC.

Summary: Parent api and for each parent there is a child api which needs to be processed.

I have a rest api pipeline which returns thousands of rows and each row has has high level information along with url field which needs to be processed to get additional information.

 

Currently I am processing parent url to snowflake. Manually downloading all url’s to a file and processing the url’s to get additional information to snowflake.

 

Is there a better approach for the process which can be automated. 

 

icon

Best answer by Bikram 5 June 2023, 22:41

View original

2 replies

Bikram
Rank
Userlevel 5
Badge +1

 

@va7777 

In this case you can fetch data from parent URL and store into a file in s3/local directory.

In the 2nd pipeline , read the file which you store in s3/local directory and pass URL to http client to get the result form respective URLs and store into snowflake db.

 

The pipeline should look like below and let me know if it helps.

 

 

 

va7777
Rank

Thank you this is a very good solution to address the issue.

Reply


Terms & Conditions