Question

Error when trying to Streamsets in Kubernetes cluster

  • 18 April 2022
  • 3 replies
  • 47 views

I’m trying to deploy Streamsets in a AWS EKS cluster with several approaches, but none of them seem to be working.

First I tried to deploy a control agent following the steps here: https://streamsets.com/blog/deploy-dataflow-pipelines-kubernetes-streamsets-control-agent/

When I run the startup script for EKS with the required parameters I receive the following message:

Running common-login.sh

Failed to authenticate with SCH :(

 

If I debug the script, I can see the following error returned by the login service:

{
"EXCEPTION": {
"rawMessage": "java.lang.IllegalArgumentException: Invalid/unsupported hash version 'UNKNOWN'",
"className": "java.lang.IllegalArgumentException",
"message": "Invalid/unsupported hash version 'UNKNOWN'",
"stackTrace": "java.lang.IllegalArgumentException: Invalid/unsupported hash version 'UNKNOWN'\n\tat com.streamsets.lib.security.http.PasswordHasher.getValueToHash(PasswordHasher.java:131)\n\tat com.streamsets.lib.security.http.PasswordHasher.verify(PasswordHasher.java:167)\n\tat com.streamsets.apps.security.backend.impl.JpaSecurityBackend$EmbeddedUserGroupProvider.authenticate(JpaSecurityBackend.java:5983)\n\tat com.streamsets.apps.security.backend.impl.JpaSecurityBackend.lambda$validateUserCredentials$27(JpaSecurityBackend.java:3192)\n\tat com.streamsets.apps.security.backend.impl.JpaSecurityBackend.doInContext(JpaSecurityBackend.java:5825)\n\tat com.streamsets.apps.security.backend.impl.JpaSecurityBackend.validateUserCredentials(JpaSecurityBackend.java:3192)\n\tat com.streamsets.apps.security.publicrest.AuthenticationResource.login(AuthenticationResource.java:68)\n\tat jdk.internal.reflect.GeneratedMethodAccessor1798.invoke(Unknown Source)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:566)\n\tat org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)\n\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)\n\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)\n\tat org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)\n\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)\n\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)\n\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)\n\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)\n\tat org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)\n\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)\n\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)\n\tat org.glassfish.jersey.internal.Errors.process(Errors.java:292)\n\tat org.glassfish.jersey.internal.Errors.process(Errors.java:274)\n\tat org.glassfish.jersey.internal.Errors.process(Errors.java:244)\n\tat org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)\n\tat org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)\n\tat org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)\n\tat org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)\n\tat org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)\n\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)\n\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)\n\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)\n\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)\n\tat org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)\n\tat org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:319)\n\tat org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:273)\n\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)\n\tat org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)\n\tat com.streamsets.apps.common.rest.CredentialsFilter.doFilter(CredentialsFilter.java:32)\n\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)\n\tat org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)\n\tat com.streamsets.domainserver.mdc.MDCFilter.doFilter(MDCFilter.java:52)\n\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)\n\tat org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)\n\tat com.streamsets.apps.common.rest.PaginationServletPreFilter.doFilter(PaginationServletPreFilter.java:227)\n\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)\n\tat org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)\n\tat com.streamsets.domainserver.webapp.DSWebAppLoader$RequestIdFilter.doFilter(DSWebAppLoader.java:531)\n\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)\n\tat org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:716)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:239)\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:516)\n\tat com.streamsets.lib.security.http.LimitedMethodServer.handle(LimitedMethodServer.java:29)\n\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)\n\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)\n\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)\n\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n"
}
}

Any idea why and how to solve it? If I use a non matching user/password I get a login error.

 

I have also tried to deploy a data collector in Kubernetes as mentioned in 

My engine is successfully connected to my Control Hub, but it still asks me to create a deployment before building a pipeline.

 

How should I proceed to have Streamsets working in my k8s environment?


3 replies

Userlevel 1

If you are on StreamSets DataOps Platform and have Legacy Kubernetes support enabled, the best way to deploy a Control Agent is using Helm as described in the docs here:

https://docs.streamsets.com/portal/platform-controlhub/controlhub/UserGuide/DataCollectorsProvisioned/ProvisionSteps.html 

 

If you are on StreamSets Control Hub v3.x, you can also use Helm to deploy a Control Agent as described here:

https://docs.streamsets.com/portal/controlhub/latest/help/controlhub/UserGuide/DataCollectorsProvisioned/ProvisionSteps.html

 

An example Helm 3.x install command is:

$ helm install  --namespace  <your namespace>  control-agent   /path/to/helm-charts/stable/control-agent

 

If you do not want to use Helm, you could use an alternative approach as described here (note there are separate sections for DataOps Platform and Control Hub 3.x):

https://github.com/onefoursix/control-agent-k8s-deployment

If you are on StreamSets DataOps Platform and have Legacy Kubernetes support enabled, the best way to deploy a Control Agent is using Helm as described in the docs here:

https://docs.streamsets.com/portal/platform-controlhub/controlhub/UserGuide/DataCollectorsProvisioned/ProvisionSteps.html 

 

If you are on StreamSets Control Hub v3.x, you can also use Helm to deploy a Control Agent as described here:

https://docs.streamsets.com/portal/controlhub/latest/help/controlhub/UserGuide/DataCollectorsProvisioned/ProvisionSteps.html

 

If you do not want to use Helm, you could use an alternative approach as described here (note there are separate sections for DataOps Platform and Control Hub 3.x):

https://github.com/onefoursix/control-agent-k8s-deployment

I’m a new user to Control Hub. I registered in the DataOps Platform and I’m trying to find the best way to set it up. Any reccommendation?

Userlevel 1

Hi @nramos , As mentioned in the docs here,  Kubernetes Support in StreamSets DataOps Platform can be enabled by request for customers with Professional or Enterprise Accounts.  Please contact your StreamSets Account Team if you are in that category.

Thank you

Mark

Reply