Environment:
- Streamsets Data Collector.
- Streamsets Library 1.X
Issue:
This has been observed that snowflake pipeline is failing with “user authentication” error when pipeline is idle for more than 4-6+ hours.
This is the error stack:
com.streamsets.pipeline.api.base.OnRecordErrorException: SNOWFLAKE_65 - File upload to Snowflake failed: net.snowflake.client.jdbc.SnowflakeReauthenticationRequest: Authentication token has expired. The user must authenticate again.
at com.streamsets.pipeline.stage.destination.snowflake.uploader.SnowflakeFileUploaderTarget.write(SnowflakeFileUploaderTarget.java:155)
at com.streamsets.pipeline.api.base.configurablestage.DTarget.write(DTarget.java:34)
at com.streamsets.datacollector.runner.StageRuntime.lambda$execute$2(StageRuntime.java:291)
at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:232)
at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:299)
at com.streamsets.datacollector.runner.StagePipe.process(StagePipe.java:209)
at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.processPipe(ProductionPipelineRunner.java:859)
at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.lambda$executeRunner$3(ProductionPipelineRunner.java:903)
at com.streamsets.datacollector.runner.PipeRunner.acceptConsumer(PipeRunner.java:195)
at com.streamsets.datacollector.runner.PipeRunner.forEachInternal(PipeRunner.java:140)
at com.streamsets.datacollector.runner.PipeRunner.executeBatch(PipeRunner.java:120)
at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.executeRunner(ProductionPipelineRunner.java:902)
at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.runSourceLessBatch(ProductionPipelineRunner.java:880)
at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.processBatch(ProductionPipelineRunner.java:503)
at com.streamsets.datacollector.runner.StageRuntime$3.run(StageRuntime.java:371)
at java.security.AccessController.doPrivileged(Native Method)
at com.streamsets.datacollector.runner.StageRuntime.processBatch(StageRuntime.java:367)
at com.streamsets.datacollector.runner.StageContext.processBatch(StageContext.java:293)
at com.streamsets.pipeline.lib.dirspooler.SpoolDirRunnable.produce(SpoolDirRunnable.java:314)
at com.streamsets.pipeline.lib.dirspooler.SpoolDirRunnable.run(SpoolDirRunnable.java:135)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:214)
at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:43)
at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:24)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:210)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeRunnable.run(SafeScheduledExecutorService.java:176)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:214)
at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:43)
at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:24)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:210)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeRunnable.run(SafeScheduledExecutorService.java:176)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: net.snowflake.client.jdbc.SnowflakeReauthenticationRequest: Authentication token has expired. The user must authenticate again.
at net.snowflake.client.jdbc.SnowflakeUtil.checkErrorAndThrowExceptionSub(SnowflakeUtil.java:121)
at net.snowflake.client.jdbc.SnowflakeUtil.checkErrorAndThrowExceptionIncludingReauth(SnowflakeUtil.java:60)
at net.snowflake.client.core.SessionUtil.tokenRequest(SessionUtil.java:855)
at net.snowflake.client.core.SessionUtil.renewSession(SessionUtil.java:775)
at net.snowflake.client.core.SFSession.renewSession(SFSession.java:575)
at net.snowflake.client.core.SFStatement.executeHelper(SFStatement.java:488)
at net.snowflake.client.jdbc.SnowflakeFileTransferAgent.parseCommandInGS(SnowflakeFileTransferAgent.java:1180)
at net.snowflake.client.jdbc.SnowflakeFileTransferAgent.parseCommand(SnowflakeFileTransferAgent.java:843)
at net.snowflake.client.jdbc.SnowflakeFileTransferAgent.<init>(SnowflakeFileTransferAgent.java:819)
at net.snowflake.client.core.SFStatement.executeFileTransfer(SFStatement.java:748)
at net.snowflake.client.core.SFStatement.executeQuery(SFStatement.java:127)
at net.snowflake.client.core.SFStatement.execute(SFStatement.java:737)
at net.snowflake.client.core.SFStatement.execute(SFStatement.java:635)
at net.snowflake.client.jdbc.SnowflakeStatementV1.executeInternal(SnowflakeStatementV1.java:287)
at net.snowflake.client.jdbc.SnowflakeStatementV1.execute(SnowflakeStatementV1.java:337)
at com.zaxxer.hikari.pool.ProxyStatement.execute(ProxyStatement.java:92)
at com.zaxxer.hikari.pool.HikariProxyStatement.execute(HikariProxyStatement.java)
at com.streamsets.pipeline.stage.destination.snowflake.uploader.SnowflakeFileUploaderTarget.write(SnowflakeFileUploaderTarget.java:152)
Solution:
This could be probably due to an idle session for 4hrs at which time the session token gets expired. The Snowflake provides parameters that let you control the behaviour of your account.
We can add set the CLIENT_SESSION_KEEP_ALIVE = true which will not force a user to log in again after a period of inactivity in the session.
https://docs.snowflake.com/en/sql-reference/parameters.html#client-session-keep-alive
