Question

Streamset Snowflake staging issue

  • 19 January 2023
  • 0 replies
  • 31 views

Hello Everyone

 

I am facing following issue. Any inputs will be helpful.

 

  1. We are using Streamsets to load data into Snowflake.
  2. Our Snowflake instance is hosted in the region [EU-WEST-1] and S3 (for Stage) is in [EU-WEST-2]
  3. We configured Snowflake connector [Staging] in the StreamSets to use S3 region same as Snowflake. With this setting in place, we see following errors getting logged in S3 bucket (which is meant to capture the errors):
"errorMessage": "SNOWFLAKE_59 - Could not get a Stage File Writer instance to write the records: 'The bucket is in this region: eu-west-2. Please use this region to retry the request",

 

  1. If we change S3 Region for the Snowflake connector in the Streamsets to [EU-WEST-2], we see following error in the Streamsets UI.
SNOWFLAKE_19 - Could not access the S3 bucket associated with the stage: com.amazonaws.SdkClientException: Unable to execute HTTP request: s3.eu-west-2.aws.amazonaws.com: Name or service not known

 

Streamset documentation says following:

 

“To stage data in an Amazon S3 external stage, create a Snowflake external stage in a bucket in the same S3 region that hosts your Snowflake virtual warehouse. For example, if your Snowflake warehouse is in AWS US West, then create the Snowflake external stage in a bucket in the AWS US West region.”

 

Does this mean there is a limitation on Streamsets side though Snowflake does not have such limitation? If yes, then is there a work-around for this please?

 

Thank you.


0 replies

Be the first to reply!

Reply