Skip to main content
Question

Pipeline load data to Snowflake table, even though the target table does not exit, pipeline still succeeded

  • 19 August 2024
  • 4 replies
  • 21 views

I created a pipeline load data from oracle to snowflake. even though the target table in snowflake does not exist in snowflake , the pipline still show succeeded without any error, also without any data loaded. 

If checked the log, it is said “Error SnowflakeSQLException: SQL compilation error

table XXXX does not exist or not authorized.

”

How to make the pipeline failed if found target table is incorrect, or column is incorrect, for such kind of normal errors reporting in data load? 

@Dolphin, perhaps it indicates success due to error record handling configuration. Can you share the configuration of your Snowflake stage and also check how that stage is configured to handle 1) errors and 2) if the target table does not exist?

@Sperch , below is my config, by the way, I found if I change the snowflake target table from parameter value to a hardcode table name, and if the target table not in snowflake, it will failed.

However if I use parameter value for snowflake target table name, then even the table not in snowflake, it will still succeeded. 

@Dolphin, if that is the case then it seems you may have in fact uncovered a bug. I would recommend that you check the release notes for your SDC version’s known bugs and if it isn’t the latest see if that bug is mentioned or even remediated in a later release. If not, then you might want to report this to StreamSets support to let them know about it.

Great troubleshooting, by the way.

@Dolphin please also verify if Table Auto Create option is enabled. If it is, then the table will be created if not exist in Snowflake

 

 

Reply


Terms & Conditions