30-Day Free Trial: It’s Never Been Easier To Get Started With StreamSets
Get inspired and gain all the knowledge you need
Recently active
Hi -Welcome to the StreamSets Academy area of Community!Please post questions related to the self-paced video course “DataOps Platform Fundamentals”. Please use the tag “DataOps Platform Fundamentals Course” when creating your question under the “StreamSets Academy” category. Thank you!
"Failed at step EXEC - Permission denied" when starting SDC as a Service on Systems with SELinux ProblemWhen starting StreamSets Data Collector as a service under systemd, the service fails immediately on startup. The following error is shown when the status of the service is checked (using the systemctl status sdc command).systemd[13801]: sdc.service: Failed at step EXEC spawning /opt/streamsets-datacollector/bin/streamsets: Permission denied CauseThis error indicates that the script SDC_HOME/bin/streamsets (which is used to start SDC) could not be launched by systemd. This can be caused by incorrect file or directory permissions or by problems with the SELinux context (on systems where SELinux is enabled). SolutionStep 1: Examine the Service ConfigurationBefore looking at possible permissions issues, it is important to verify that the user, group, and installation directory for the service are correctly configured in the service unit file, /etc/systemd/system/sdc.service. Exami
When writing to Kafka Producer destination it doesn’t always handle timeout exceptions (as shown below) and the pipeline does not honor On Record Error » Send to Error setting on the Kafka Producer destination. How can this be resolved?Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. at org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.<init>(KafkaProducer.java:1186) at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:880) at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:803) at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:690) at com.streamsets.pipeline.kafka.impl.BaseKafkaProducer09.enqueueMessage(BaseKafkaProducer09.java:64) at com.streamsets.pipeline.stage.destination.kafka.KafkaTarget.writeOneMessagePerRecord(KafkaTarget.java:242) ... 30 more Caused by: org.apache.kafka.common.errors.TimeoutExc
Problem DescriptionWhen running an SDC pipeline which processes records with Strings containing UTF-8 special characters, these special characters are being replaced by question marks (? or �) in various parts of your pipeline.ExampleInput record:{"productName": "My Product™"}Output record:{"productName": "My Product�"}Root CauseThis problem indicates that your Java Runtime Environment is using a non-UTF-8 character set which does not support these special characters, so the special characters are being replaced by the � (U+FFFD) REPLACEMENT CHARACTER.By default, SDC will try to set your JAVA_OPTS to use the UTF-8 encoding, however it is possible that the JAVA_OPTS parameters are being set outside of SDC and are overriding either of the following JVM parameters which tells the JRE what encoding to use at runtime: file.encoding and sun.jnu.encoding.The JRE also has a default character set which it will use if these two parameters are not specified. This default character set can vary be
I'm trying to enable Kerberos for my SDC RPM installation, but when I start the SDC I get following exception:java.lang.RuntimeException: Could not get Kerberos credentials: javax.security.auth.login.LoginEx Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:897) at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760) at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498)How do I move forward?
Trying to read XML in the following format with “:” is throwing Can't parse XML element names containing colon ':' error.<sh:root> <sh:book> </sh:book> <sh:genre> </sh:genre> <sh:id> </sh:id> <sh:book> </sh:book> <sh:genre> </sh:genre> <sh:id> </sh:id> <sh:book> </sh:book> <sh:genre> </sh:genre> <sh:id> </sh:id></sh:root>What’s the best way to read such XML? (Note that changing “:” to “_” in the XML works.)
I am trying to write to Amazon S3 destination with its Authentication Method set to AWS Keys, but when I run the pipeline I get “Unable to write object to Amazon S3: The request signature we calculated does not match the signature you provided. Check your key and signing method.” error.Here’s the entire stack trace…Caused by: com.streamsets.pipeline.api.StageException: S3_21 - Unable to write object to Amazon S3, reason : com.amazonaws.services.s3.model.AmazonS3Exception: The request signature we calculated does not match the signature you provided. Check your key and signing method. (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: 12345678915ABCDE; S3 Extended Request ID: xyzxyzxyzxyzxyzxyz=; Proxy: null), S3 Extended Request ID: xyzxyzxyzxyzxyzxyz=at com.streamsets.pipeline.stage.destination.s3.AmazonS3Target.write(AmazonS3Target.java:182)at com.streamsets.pipeline.api.base.configurablestage.DTarget.write(DTarget.java:34)at com.streamsets.datacoll
I'm running SDC in a Docker container and I am trying to invoke a HTTP call with HTTP Client using a restricted header like below:curl -X GET -H "Host: my-host" http://my-service/my-endpoint However, by default the header is ignored with the following warning:HttpUrlConnector - Attempt to send restricted header(s) while the [sun.net.http.allowRestrictedHeaders] system property not set. Header(s) will possibly be ignoredWhere do I set this property? Adding it to SDC config did not help.
IssueA pipeline fails due to a Kafka Stage with Kerberos authentication enabled is throwing the stage exception KAFKA_29 - Error fetching data from Kafka: org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: xxxxBackgroundThere are two general types of errors which can happen which prevent a user from connecting to a secured endpoint: Authentication and Authorization.Authentication errors - have to do with a credential failures, i.e. a Kerberos ticket is unable to be granted due to the wrong credentials being used, denying the principal from obtaining a service ticket. Authorization errors - occur after the user has made it passed authentication, but there is some additional Access Control Logic (ACL) which is preventing this user from accessing some resource (i.e. missing permissions to access a certain Kafka Consumer group).ResolutionThe GroupAuthorizationException signifies that the Kafka consumer has successfully authenticated and obtained a s
Getting this error While create JDBC connection for GreenPlum Database:JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: [Pivotal][Greenplum JDBC Driver][Greenplum]no pg_hba.conf entry for host "172.18.0.2", user "sdc", database "postgres", SSL off.Using Greenplum 9.5 in docker.
I have a StreamSets Data Collector running in Docker and when I run a pipeline with Kafka Consumer I am seeing these error messages:The configuration = was supplied but isn't a known configThe configuration schema.registry.url = was supplied but isn't a known config How do I get past this error?
while running python program using streamsets python sdk we are getting error File "/var/lib/jenkins/.local/lib/python3.8/site-packages/streamsets/sdk/exceptions.py", line 5, in <module> from requests.exceptions import HTTPError, JSONDecodeError ImportError: cannot import name 'JSONDecodeError' from 'requests.exceptions'we have found solution for this but we want to know why we are getting this error.Solution for this is manually we are replacing line “from requests.exceptions import HTTPError, JSONDecodeError” with “from json import JSONDecodeError”
Empty values are loaded as NULLs when loading the data into a snowflake table. For example, I have an input field of varchar type and sometimes the data in it is empty(‘’). I would want the input data ‘’ to be loaded into snowflake same as ‘’ and not NULL. This will be helpful in cases where I have defined the destination table field as NON NULL.
ENV:-SDC is integrated with CDH/CDP. Kerberos is enabled. start event script/end event scripts are being used to get the kerberos credentials.ISSUE:-start event script/end event scripts try to kinit and end up with following error message:-ERROR stderr: kinit: Pre-authentication failed: Permission denied while getting initial credentials Resolution:-By default CDH/CDP enabled clusters has set environment variable KRB5CCNAME add following line to at the beginning of the script to unset the variable temporarilyunset KRB5CCNAME
Is there a more sophisticated way of removing backslash within a string? I've tried string.replace(/\\/g, ""), but that did not help.
Thread starvation or clock leap can be caused due to different causes but mainly it happens because of below reasons-High load on the engine The server is so busy that thread starvation is occurring, which is preventing the housekeeper thread from running. GC running for a long time. Insufficient resources (heap or CPU).Possible solutions for this issue could be the following: Increasing the memory of the engine. Rebalancing the jobs running on the engine. Dividing a big pipeline into multiple parts. Reducing the Batch size
IssueA Kafka stage fails to connect to the broker due to the follow error stack trace:INFO ConsumerCoordinator - (Re-)joining groupINFO NetworkClient - Disconnecting from node xxx due to socket connection setup timeout. The timeout value is 8424 ms.INFO ConsumerCoordinator - Group coordinator is unavailable or invalid due to cause: null. isDisconnected: true. Rediscovery will be attempted.2023-09-27 13:07:58,776 INFO ConsumerCoordinator - Request joining group due to: rebalance failed due to 'null' (DisconnectException)BackgroundThis specific Kafka Client exception appears when the Kafka Client’s socket-connection-setup-timeout-max-ms is set too low, causing it to attempt to re-join the group.The maximum amount of time the client will wait for the socket connection to be established. The connection setup timeout will increase exponentially for each consecutive connection failure up to this maximum. SolutionWhen this property is set too low, the Kafka Client will time out during the
My JDBC producer stage is giving the error java.lang.NoClassDefFoundError: org/apache/commons/lang3/StringUtils Has anyone had a similar issue before?
I’m running below command in my windows machine for Auth token and getting below error message. curl -s -X POST -d '{"userName":"userid", "password": "password"}' https://cloud.streamsets.com/security/public-rest/v1/authentication/login -H "Content-Type:application/json" -H "X-Requested-By:SCH" Invoke-WebRequest : Cannot bind parameter 'Headers'. Cannot convert the "Content-Type:application/json" value of type "System.String" to type "System.Collections.IDictionary".At line:1 char:168+ ... st/v1/authentication/login -H "Content-Type:application/json" -H "X-R ...+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : InvalidArgument: (:) [Invoke-WebRequest], ParameterBindingException + FullyQualifiedErrorId : CannotConvertArgumentNoMessage,Microsoft.PowerShell.Commands.InvokeWebRequestCommand Can you please help?
Question:How to connect JDBC Consumer Stage to MS SQL Server via AAD or Active Directory? Answer:Our product teams are currently evaluating to see this could be added natively. Also, our Product Teams has this on the future Product Road Map for native support.For now you will get the following error message if you try to use “authentication=ActiveDirectoryPassword” in your JDBC URL.JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Failed to load MSAL4J Java library for performing ActiveDirectoryPassword authentication.: hikariConfigBean.connectionStringThis is because the out of box Microsoft SQL Server Drivers comes directly from Microsoft. Microsoft does not package support directly for the authentication method ActiveDirectoryPassword. For this to work you must acquire MSAL4J and it’s dependencies and add them to SDC’s java classpath for the given JDBC Stage. You can add the jars via the pipelin
I have a few JDBC-based stages in my pipeline (Origin, JDBC Lookup, etc.) and when I try to replace the existing JDBC-based origin with another (for example, Oracle CDC with MySQL Binary Log), the validation just on the lookup processors fails with “Failed to get driver instance with multiple JDBC connections” error. Even though I haven’t changed anything on those processors.Here’s the stack trace…java.lang.RuntimeException: Failed to get driver instance for jdbcUrl=jdbc:oracle:thin:@connection_URL at com.zaxxer.hikari.util.DriverDataSource.<init>(DriverDataSource.java:112) at com.zaxxer.hikari.pool.PoolBase.initializeDataSource(PoolBase.java:336) at com.zaxxer.hikari.pool.PoolBase.<init>(PoolBase.java:109) at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:108) at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:81) at com.streamsets.pipeline.lib.jdbc.JdbcUtil.createDataSourceForRead(JdbcUtil.java:875) at com.streams
Hello, could you please help me out?In Transformer pipelines adding a Kafka consumer throwsjava.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.sql.kafka010.consumer.KafkaDataConsumer$The same connection works fine with Data Collector.I am using local Spark in docker Thanks!
Issue:The SDC is showing following error while start the process on sone of the java version however this will not stop the data collector process from booting. This can be seen while booting only.# WARNING: Unable to get Instrumentation. Dynamic Attach failed. You may add this JAR as -javaagent manually, or supply -Djdk.attach.allowAttachSelf# WARNING: Unable to attach Serviceability Agent. Unable to attach even with module exceptions: [org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.openjdk.jol.vm.sa.SASupportException: Sense failed.] Solution:There are 2 ways to suppress the warning message:We can added the global variable in linux which will enable this property for all the java process running on linux machine, however the JAVA_TOOL_OPTIONS variable will be unset after the reboot. If you want to enable it, for SDC process only then follow the second solution. export JAVA_TOOL_OPTIONS="-Djdk.attach.allowAttachSel
Hi, Im planning to take up exams for Streamset whitebelt certification in the coming days. Is there any inputs that anyone can provide around how to attempt for the same?
Hello,I am using a Field Type Converter to convert string to datetime with date format (yyyy-MM-dd'T'HH:mm:ss.SSS'Z' - 2021-08-18T08:20:24.136000Z). However, the same field has dates with format 2021-08-18T08:20:24.00Z, which it gives me error CONVERTER_00 - Failed to convert field '/createdate' of type 'STRING' with value '2021-08-19T14:01:32Z' to type 'DATETIME' Any recommendations on how to handle two different date formats for the same field?
