Question

About StreamSets to load data to Snowflake table

  • 23 September 2022
  • 3 replies
  • 54 views

Hi team,

We are using streamsets to load data to snowflake table. We configured an common Internal Stage for the pipeline, as well as target tables, can we understand the mechanism for such pipelines are firstly load data as file and put into Snowflake cloud (Internal Stage), and then will trigger a copy command from Internal Stage to taget Snowflake tables like:
copy into mytable … (is it from user stage or table stage or name stage)?

Can StreamSets load data to external stage and then copy to table?

ANy cost difference for load data to external stage and internal stage by StreamSets?
I suppose it is just the cost difference between the storage cost of snowflake cloud and external cloud, cause external cloud a bit cheaper than snowflake internal cloud? Any computing resource difference when loading to internal and external stages. 

Thanks. 

 


3 replies

Userlevel 3
Badge

Hello @Dolphin ,

Thank you for reaching out to us.

The Snowflake destination stages files to either an internal Snowflake stage or an external stage in Amazon S3 or Microsoft Azure. Then, the destination sends a command to Snowflake to process the staged files. 

Here is the link: 

https://docs.streamsets.com/portal/datacollector/5.0.x/help//datacollector/UserGuide/Destinations/Snowflake.html#task_nfs_c2k_mfb

 

 

Thank you,

Akshay Jadhav
 

Userlevel 3
Badge

@Dolphin - Please mark the comment as “best answer” if the suggestion is helpful.

 

Userlevel 3
Badge

@Dolphin - Please mark the comment as “best answer” if the suggestion is helpful.

Reply