Snowflake stage validation error: java.lang.NoSuchFieldError: CLIENT_INFO

After upgrading to a newer Transformer version, you might face an error validating your pipeline with a Snowflake stage with a message similar to: java.lang.NoSuchFieldError: CLIENT_INFO.
 

The full stack trace is the following:

[stage:] ERROR DataTransformerLauncher - Error stack trace:
java.lang.NoSuchFieldError: CLIENT_INFO
	at net.snowflake.spark.snowflake.JDBCWrapper.getConnector(SnowflakeJDBCWrapper.scala:207)
	at net.snowflake.spark.snowflake.Utils$.getJDBCConnection(Utils.scala:277)
	at com.streamsets.pipeline.spark.destination.snowflake.SnowflakeDestination.init(SnowflakeDestination.scala:98)
	at com.streamsets.datatransformer.dag.BaseBatchDAGRunner$$anonfun$2.apply(BaseBatchDAGRunner.scala:303)
	at com.streamsets.datatransformer.dag.BaseBatchDAGRunner$$anonfun$2.apply(BaseBatchDAGRunner.scala:301)
	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
	at scala.collection.mutable.HashMap$$anon$2$$anonfun$foreach$3.apply(HashMap.scala:139)
	at scala.collection.mutable.HashMap$$anon$2$$anonfun$foreach$3.apply(HashMap.scala:139)
	at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236)
	at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
	at scala.collection.mutable.HashMap$$anon$2.foreach(HashMap.scala:139)
	at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
	at com.streamsets.datatransformer.dag.BaseBatchDAGRunner.generateDAG(BaseBatchDAGRunner.scala:301)
	at com.streamsets.pipeline.spark.dag.SparkBatchDAGRunner.generateDAG(SparkBatchDAGRunner.scala:32)
	at com.streamsets.datatransformer.dag.runner.DataTransformerRunner.startDataTransformerDagRunner(DataTransformerRunner.java:433)
	at com.streamsets.datatransformer.dag.runner.DataTransformerRunner.start(DataTransformerRunner.java:262)
	at com.streamsets.datacollector.execution.runner.common.AsyncRunner.lambda$start$3(AsyncRunner.java:150)
	at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:214)
	at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:44)
	at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:25)
	at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:210)
	at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:214)
	at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:44)
	at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:25)
	at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:210)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at com.streamsets.datacollector.metrics.MetricSafeScheduledExecutorService$MetricsTask.run(MetricSafeScheduledExecutorService.java:88)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

Once the configuration is verified, it is likely the engine is using an old version of the Snowflake stage library.

You have to carefully look into your external libraries for any file matching the pattern snowflake-jdbc-<version>.jar. The external libraries are usually pointed by the environment variable $STREAMSETS_LIBRARIES_EXTRA_DIR.

If you find any library matching the above pattern, you can compare with the version provided by Transformer installation at $TRANSFORMER_DIST/streamsets-libs/streamsets-spark-snowflake-lib/lib/ and you should see it is a different version.

This error was reported on a real case and that is what we found on the engine directories:

$STREAMSETS_LIBRARIES_EXTRA_DIR/streamsets-spark-jdbc-lib/lib/snowflake-jdbc-3.9.2.jar

$TRANSFORMER_DIST/streamsets-libs/streamsets-spark-snowflake-lib/lib/snowflake-jdbc-3.13.14.jar

The solution is simply to remove or move away the conflicting old version library jar.

As an advice, check always the external libraries after upgrading and look for any possible conflicting library if your pipeline is not validating. You can check more information about external libraries in our documentation.