Skip to main content

Can you please help with how I can address this problem. I am using SDC.

Summary: Parent api and for each parent there is a child api which needs to be processed.

I have a rest api pipeline which returns thousands of rows and each row has has high level information along with url field which needs to be processed to get additional information.

 

Currently I am processing parent url to snowflake. Manually downloading all url’s to a file and processing the url’s to get additional information to snowflake.

 

Is there a better approach for the process which can be automated. 

 

 

@va7777 

In this case you can fetch data from parent URL and store into a file in s3/local directory.

In the 2nd pipeline , read the file which you store in s3/local directory and pass URL to http client to get the result form respective URLs and store into snowflake db.

 

The pipeline should look like below and let me know if it helps.

 

 

 

Thank you this is a very good solution to address the issue.

Reply


Terms & Conditions