Question

Download and process zip with multiple files

I need to process a zip file hosted on a website. This zip file has multiple csv files of different formats along with some pdf files.

I need to process csv files into different tables in snowflake. how can this request be achieved.

+1

can you please try to construct your pipeline like below .

In this keep the .zip in s3 bucket and send it to local FS and by using the shell try to unzip it and store it in a separate folder.

In the second pipeline you can fetch all csv files and send it to snowflake tables as per your need.

In this case you can create orchestration pipeline which will help in file processing in a robust way.

Kindly let me know if it helps.

#! /bin/sh

unzip -o *.zip -d /DataAfterUnzipped

Thank you @Bikram This is definitely a workable solution. I will test it out. Thanks,

Sign up