Can you please help with how I can address this problem. I am using SDC.
Summary: Parent api and for each parent there is a child api which needs to be processed.
I have a rest api pipeline which returns thousands of rows and each row has has high level information along with url field which needs to be processed to get additional information.
Currently I am processing parent url to snowflake. Manually downloading all url’s to a file and processing the url’s to get additional information to snowflake.
Is there a better approach for the process which can be automated.
