Skip to main content
Question

Unable to run snowflake multiple queries in Parallel using Snowflake Executor

  • July 22, 2022
  • 3 replies
  • 832 views
Anupam Kushwah
Fan

Hi All,

 

I am trying to run multiple queries in parallel using snowflake executor. I have set the property to run all the queries in parallel. but still on snowflake it is running one at a time.

 

also increased connection pool size to 10.

 

Thanks,

Anupam

3 replies

alex.sanchez
StreamSets Employee
Forum|alt.badge.img
  • alex.sanchezStreamSets Employee
  • StreamSets Employee
  • 73 replies
  • July 25, 2022

Hi @Anupam Kushwah,

You are using the JDBC Executor to execute those, which is not supported, we release the Snowflake Executor which takes care of some specifics of Snowflake.


Sadly, it is currently disabled since it is not possible to submit multiple queries to Snowflake in a performant way and keeping track of them as we need.

Thanks

Alex - StreamSets Engineering Manager @ Collector
Anupam Kushwah
Fan

Thanks Alex, is there any way we can run snowflake queries in parallel?

alex.sanchez
StreamSets Employee
Forum|alt.badge.img
  • alex.sanchezStreamSets Employee
  • StreamSets Employee
  • 73 replies
  • July 26, 2022

If you are using a multithreaded origin, each thread will create instances of the rest of the stages, so you will virtually have multiple instances of the Snowflake Executor

Alex - StreamSets Engineering Manager @ Collector


Terms & ConditionsAccessibility statement