Skip to main content

Community Articles and Got a Question?

Can't find what you're looking for? Ask it here or check out the Community articles

  • 1,026 Topics
  • 2,315 Replies
1026 Topics
paul
StreamSets Employee
paulStreamSets Employee
posted in Community Articles and Got a Question?
Changing Garbage Collection settings for a Data Collector engine on Platform

QuestionHow to change the Garbage Collector settings for a Data Collector engine on StreamSets Platform?AnswerPlease refer to the Java Configuration Options topic in the latest Platform documentation for instructions. Use the settings listed in the Garbage Collector section.This procedure differs from the method used to change Java configuration on Legacy Data Collector. Do not modify files directly on the Data Collector engine in Platform; instead use the UI as explained in the documentation. NOTE: Setting the Garbage Collector to G1GC is only supported for Java 8. If you use Java 11 or 17, applying  these settings will result in the JVM being unable to start due to an invalid Java configuration option being specified.

0
paul
StreamSets Employee
1 year ago
paul
StreamSets Employee
paulStreamSets Employee
posted in Community Articles and Got a Question?
Cloudera CDP 7.1.x with Kerberos - Hive Metastore destination stage library error

IssueUnable to use Hive Metastore destination in a Cloudera CDP 7.1.x environment with Kerberos.A stage library error occurs on the Hive Metastore destination stage. This does not occur with previous (6.3) version of the stage library.Environment Cloudera CDP 7.1.x Hive Metastore destination Stage library: CDP 7.1 Data Collector v4.x and 5.x, registered with Control Hub ‘Hadoop Impersonate current user’ option is enabled Root CauseWhen Data Collector is registered with Control Hub, authentication occurs with a different username in the format user@organization instead of the full username user@domain.com.In CDP 7.1.x, Cloudera made a change. Previously, a warning message would be printed in the log, but the stage library was not blocked, so the stage worked as expected. In CDP 7.1.x, an exception is thrown instead, which causes the stage library error.ResolutionChange the dpm.alias.name.enabled configuration option to true. More information on this setting can be found in the

0
paul
StreamSets Employee
1 year ago
upasana
Fan
upasanaFan
asked in Community Articles and Got a Question?
Merge two records in Jython evaluator based on particular field

Suppose, I am getting 4 records in jython evaluator stage. I want to combine them and make them 2 records based on particular field.  {name: “Henry”,age: 12}{name : ”Henry”,emai: “Henry@gmail.com} The record should become like {name: “Henry”,age: 12,emai: “Henry@gmail.com} similarly if there 2 more records like that should combine separately. 

0
upasana
Fan
1 year ago
nicholas.mendoza
Fan
nicholas.mendozaFan
posted in Community Articles and Got a Question?
OutOfMemory - Compressed class space errors

 ProblemData Collector encountering OutOfMemory error: java.lang.OutOfMemoryError: Compressed class spaceThis error occurs when the capacity for classes and their metadata is exceeded. When the flag UseCompressedClassPointers is enabled, which it is by default, the capacity for class metadata is fixed at the amount set by CompressedClassSpaceSize. This value is 1G by default. SolutionExplicitly set the JVM option -XX:CompressedClassSpaceSize to a larger amount than 1G.Ex:export SDC_JAVA_OPTS="-XX:CompressedClassSpaceSize=2G ${SDC_JAVA_OPTS}" 

0
nicholas.mendoza
Fan
1 year ago
Lavender
StreamSets Employee
LavenderStreamSets Employee
posted in Community Articles and Got a Question?
Error when connecting SDC to Denodo application using JDBC connection

Error: JDBC_00 - Cannot connect to specified database: java.sql.SQLException: Connection error: Check the host name and port number are correct, and that client and server belong to the same Denodo major version. Cannot trust the server to establish a SSL connection when connecting pipeline to Denodo using JDBC.Analysis: This is the issue with SSL certificate configuration. This might occur to due SSL misconfiguration or SDC not enabling SSL connection.Solution: You can first try to turn on SSL at the pipeline level by adding "?ssl=true" in the end of your connection string.For example, it could look like this:jdbc:vdb://<host>:<port>/database_name?ssl=trueIf the error still persists, please double check your SSL configuration to see if the certificate you imported to the truststore is valid, of if you have correctly pointed SDC to the right truststore.Below is the process you can use to extract Denodo cert and import to truststore:Use the following command to extract the c

0
Lavender
StreamSets Employee
1 year ago
Lavender
StreamSets Employee
LavenderStreamSets Employee
asked in Community Articles and Got a Question?
Configuring Snowflake Key Pair authentication without using User Credentials

Issue: In SDC 5.5.0 and below, when configuring Snowflake stage, user is still forced to configure username and password even if key pair (private_key_file and private_key_file_password) are used in the Snowflake Connection Properties. This is due to the current implementation of the stage always requires username and password.  Solution: In SDC 5.6.0 and above, in Snowflake Connection Info tab, user can now choose to use Key Pair to connect to Snowflake without being prompted for User Credentials. Selecting Key Pair will allow the user to authenticate using Public Key and Private Key to make the connection to Snowflake.User can follow this Snowflake documentation https://docs.snowflake.com/en/user-guide/key-pair-auth to create a key pair and add Private Key path to pipeline configuration. To generate the keys, please use these following commands: For encrypted private key: openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -v1 PBE-SHA1-3DES -out rsa_key.p8 For non-encrypted priva

0
Lavender
StreamSets Employee
1 year ago
som
Fan
somFan
posted in Community Articles and Got a Question?
HTTP Client supports data-urlencode values to be passed via OAUTH 2

ISSUE:While passing data-urlencode values via OAUTH 2 in HTTP client, We were getting error message-”HTTP_01 - Error fetching resource. HTTP-Status: 400 Reason: Bad Request”.The endpoint is microsoftonline oauth2. When we run below query in postman it works.curl --location --request GET 'https://login.microsoftonline.com/<xxx>/oauth2/v2.0/token \--header 'Content-Type: application/x-www-form-urlencoded' \--data-urlencode 'grant_type=client_credentials' \--data-urlencode 'client_id=<stripped>' \--data-urlencode 'client_secret=<stripped>' \--data-urlencode 'scope=https://xxx.com/.default'  but this is not working when we do this in HTTP client with below configuration.Configuration:Error Message:HTTP_01 HTTP_01 - Error fetching resource. HTTP-Status: 400 Reason: Bad Request  Resolution:Do add below property in HTTP  accept:*/*     We might be getting few error which can be fixed by following below process Error-"HTTP_00 - Cannot parse record. HTTP-Status:200 cause

0
som
Fan
1 year ago
H
Fan
harshit_agrawal1Fan
asked in Community Articles and Got a Question?
Shell Executer error: "stderr: sudo: unable to initialize policy plugin"! and one more...

Hi I am trying to use custom scripts to fetch the directory tree from the shell executor While running the script I am facing this error:“stderr: sudo: unable to initialize policy plugin”Another error which I am facing is:“error record: SHELL_003 - Script return non-zero value: 1”Though i have explicitly mentioned “exit 0” at the end of the script.I have uncommented the configration in deployemnt:stage.conf_com.streamsets.pipeline.stage.executor.shell.impersonation_mode=CURRENT_USER Is there any proper syntax which I need to use to make it run or is there something that I am missing? 

0
H
Fan
1 year ago
himanshu1234567
Discovered Fame
himanshu1234567Discovered Fame
asked in Community Articles and Got a Question?
using groovy_scripting reading data from aws s3 bucket.

Hi Team,We are working on groovy script to connect to s3 bucket and read json file and convert data into stramsets records. We are able to make connection and convert json file into stream object but we are not able to convert stream object into stramsets records. Pls help us.I am providing code snippet. 

0
himanshu1234567
Discovered Fame
1 year ago
som
Fan
somFan
posted in Community Articles and Got a Question?
How to pass json file as parameter in RESTAPI call

ISSUE:We would like to pass json file as a parameter in the REST API Call to update an ACL.API I want to use is - /jobrunner/rest/v1/sdc/{sdcId}/acl.Solution: 1- Login to SCH security app:curl -X POST -d '{"userName":"DPMUserID", "password": "DPMUserPassword"}' https://cloud.streamsets.com/security/public-rest/v1/authentication/login --header "Content-Type:application/json" --header "X-Requested-By:SDC" -c cookie.txt2. Generate auth token from the security app:sessionToken=$(cat cookie.txt | grep SSO | rev | grep -o '^\S*' | rev)echo "Generated session token : $sessionToken"After that you can call the SCH REST API using the auth token: Command to fetch the Details of acl before updating:(Please change the sdc ID as per your sdc ID)curl -X GET https://cloud.streamsets.com/jobrunner/rest/v1/sdc/<sdc_id>/acl --header "Content-Type:application/json" --header "X-Requested-By:SCH" --header "X-SS-REST-CALL:true" --header "X-SS-User-Auth-Token:$sessionToken" -i Update command:curl -X PO

0
som
Fan
1 year ago
A
StreamSets Employee
ajinkyaStreamSets Employee
asked in Community Articles and Got a Question?
Why preview in Transformer is slow compared to Data Collector?

Preview in StreamSets Transformer is expected to be slow compared to StreamSets Data Collector as Transformer submits the actual job to the spark and fetch the values without writing them to the destination. Review the documentation for the more options.  

0
A
StreamSets Employee
1 year ago
A
StreamSets Employee
ajinkyaStreamSets Employee
asked in Community Articles and Got a Question?
JDBC_06 - Failed to initialize pool: Communications link failure

Issue:While using JDBC stages pipelines are failing with following error:-JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Communications link failureCaused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)Resolution:add ?enabledTLSProtocols=TLSv1.2 in the JDBC URL. Cause:The source and destination targets couldn’t agree on the cipher suites or the protocol to use for the ssl communication. 

0
A
StreamSets Employee
1 year ago
Lavender
StreamSets Employee
LavenderStreamSets Employee
posted in Community Articles and Got a Question?
How to write to multiple tables in Hive or PostgreSQL Metadata Processor

 Issue: You want to write to multiple tables in Hive or PostgreSQL Metadata Processor instead of defining a specific table name.Solution: When processing data from either of the JDBC origins, you can use a record header attribute for the table name. The JDBC origin writes the originating table name in the jdbc.tables record header attribute. If you want to write records to tables of the same name, you can use ${record:attribute('jdbc.tables')} for the table name property.Below is an example in Hive Metadata Processor.Link to the documentation: https://docs.streamsets.com/portal/datacollector/5.3.x/help/datacollector/UserGuide/Solutions/JDBC_DriftSyncSolution.html#concept_s2c_qht_pcb

0
Lavender
StreamSets Employee
1 year ago
john.mcavoy
StreamSets Employee
john.mcavoyStreamSets Employee
asked in Community Articles and Got a Question?
Control Hub On-Prem `dev/02-initsecurity.sh` does not generate auth tokens

BackgroundWhen initializing a new SCH instance, the SCH’s app components need to authenticate via auth-tokens which are generated by and stored in the security database. The dev script `dev/02-initsecurity.sh` will generate auth-token.txt files in the $DPM_CONF directory for each SCH app component, and insert these auth tokens into the security database.https://docs.streamsets.com/portal/controlhub/latest/onpremhelp/controlhub/UserGuide/Install/SetUpHA.html#task_nl5_z4c_m1bIssueWhen upgrading or adding a new HA SCH instance to an existing SCH database, executing `dev/02-initsecurity.sh` results in empty auth-token.txt files are created the $DPM_CONF directory. This causes errors during startup because none of the SCH apps are able to authenticate without an auth-token.SolutionThis issue can happen if any of the following default values for environment variables used in the `dev/02-initsecurity.sh` script are incorrect:DPM_DIST=${DPM_DIST:-$(pwd)}DPM_CONF=${DPM_CONF:-$DPM_DIST/etc}DPM_A

0
john.mcavoy
StreamSets Employee
1 year ago
Douglas R
Fan
Douglas RFan
posted in Community Articles and Got a Question?
Control Hub Admin Tool does not properly load (only the banner loads)

Problem: After a fresh installation of Control Hub, the Admin Tool (http://<host name>:18632/)  does not load properly, as shown in Figure 1 below.  The banner at the top of the page loads, but the remainder of the page does not.  Note: More information on the Admin Tool can be found at: https://docs.streamsets.com/portal/controlhub/latest/onpremhelp/controlhub/UserGuide/Administration/AdminApp.html#concept_hym_pmr_h1b Figure 1: Control Hub Admin Tool page improperly loaded In this event, a warning will also be logged (in dpm.log), which contains the following (along with a stack trace): 2023-03-30 15:41:19,847 [requestId:] [app:] [componentId:] [user:] [thread:adminWebServer-122] WARN  HttpChannel - /rest/v1/system/componentsjavax.servlet.ServletException: javax.servlet.ServletException: javax.ws.rs.ProcessingException: URI is not absoluteCauseThis is caused by the endpoint URLs not having been correctly populated in $DPM_CONF/common-to-all-apps.properties during the Control Hub

0
Douglas R
Fan
1 year ago
P
Opening Band
pkandraOpening Band
asked in Community Articles and Got a Question?
Handling Kafka Tombstone Records

I have a pipeline using the Kafka multi-topic consumer as the origin and ultimately writing to Databricks.  How do I handle processing tombstone messages from Kafka?  I’m getting the following error from the origin for those records:KAFKA_74 - Tombstone message: payload is nullIs there a connection configuration property or something that will allow these records to be processed? 

0
P
Opening Band
1 year ago
P
Fan
priyankarayalaFan
asked in Community Articles and Got a Question?
JDBC Mutlitable consumer converting varbinary data to byte_array[] causing data issue.

Due to this conversion of varbinary to byte_array[] my data is getting distorted. How can I get back the original data i.e., the ASIS varbinary data to be loaded.In SQL server - the data looks like 0xFFD8FFE000104A464946000…………..In Streamsets - It read the data like /9j/4AAQSkZJRgABAQAAAQABAAD// ………………  

0
P
Fan
1 year ago
K
Fan
kushal_upadhyayFan
asked in Community Articles and Got a Question?
Writing binay file with 0 bytes in S3 bucket when binary format is selected

Send data from KAFKA To OBS (Object storage cloud customized version of AWS S3) in binary format is creating a file with 0 bytes though we have 36 records.The records from Kafka has a format of SDC records. Please help how we can write binary data to S3 bucket when format,

0
K
Fan
1 year ago
R
Fan
ravitejakshettyFan
asked in Community Articles and Got a Question?
How to query BigQuery table in the pipeline and use the result for further processing

Hi Friends,I am new to StreamSets and trying to build a pipeline to handle event data coming from Google Cloud pub/sub to Google Cloud BigQuery. I am stuck with a problem and need your help.Whenever an event enters the pipeline, I will have to check in the Bigquery tables whether the event already exists in the tables. We have a unique id in the event against which i can query the BigQuery table to check and further continue in the pipeline. I tried various options(including jdbc lookup) but no luck.Can you please help? Thanks in Advance.Regards,Ravi 

0
R
Fan
1 year ago
som
Fan
somFan
posted in Community Articles and Got a Question?
Disable fail over by rest api call in sch.

ISSUE:When we create a job in rest api, by default it creates the job with fail over enabled.Sometimes we need to create the job via rest api  with fail over disabled.we can follow below solution to achieve this.  SOLUTION:While creating the job, we can change the value of field "migrateOffsets" to “false” and it will create the job with fail over disabled.  

0
som
Fan
1 year ago
som
Fan
somFan
posted in Community Articles and Got a Question?
Control the number of email alerts per job failure.

ISSUE:When the job gets failed, we get many number of email alerts.Below is the subscription set at our end.We have also tried changing the subscription to “max global fail over retries” but it does not give actual error message and gives a generic error,{{"FROM_STATUS":"INACTIVE", "TO_COLOR":"RED", "TO_STATUS":"INACTIVE", "ERROR_MESSAGE":"Job is inactive due to global failover retries exhausted." } }Can we control the number of email alerts with getting actual error message. SOLUTION:We can control the email alert by setting below subscription where we will get only 1-2 email alert when either jobs gives any error or exception and also can capture actual error message.${TO_STATUS=="INACTIVE" && TO_COLOR=="RED" && str:contains(ERROR_MESSAGE,"RUN_ERROR") || str:contains(ERROR_MESSAGE,"exception")}  

0
som
Fan
1 year ago
J
Opening Band
jyuanOpening Band
asked in Community Articles and Got a Question?
How to get statistics like bytes streamed per job per day?

Is there a way to gather statistics on jobs?I am interested in showing the bytes streamed by each job in the last 24 hours, and the like. Also, if possible, I would like to retrieve the bytes streamed in the last 30 days. From the ControlHub GUI, I don’t see anything. Maybe the solution is available on the SDC side? Maybe we can run a script on the SDC server to gather statistics?Thanks!

0
J
Opening Band
1 year ago
A
StreamSets Employee
ajinkyaStreamSets Employee
asked in Community Articles and Got a Question?
How to get a specific pipeline using pipeline id or pipeline name using python SDK?

sch.pipelines.get(pipeline_id='id of pipeline') function can be used to get the specific pipeline using the pipeline id.  from streamsets.sdk import ControlHubserver_url = 'https://cloud.streamsets.com/'sch = ControlHub(server_url='https://cloud.streamsets.com/',username = 'username@org',password = 'password')desiredpip=sch1.pipelines.get(pipeline_id='00a2a5e6-5125-4477-b022-5507c7f63c59:dpmsupport')desiredpip=sch1.pipelines.get(name='SDC500 - Dev to Trash (batch size 10000)')NOTE:- Fetching a pipeline using pipeline id is much faster than the pipeline name.

0
A
StreamSets Employee
1 year ago
Douglas R
Fan
Douglas RFan
posted in Community Articles and Got a Question?
MySQL: JDBC_06: Failed to Initialize Connection Pool: Access Denied Error

MySQL: JDBC_06: Failed to Initialize Connection Pool: Access Denied Error ProblemA pipeline which is using a JDBC stage to access a MySQL database generates an error int he following form:START_ERROR: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Access denied for user 'username'@'hostname' (using password: YES)The pipeline is using a configured Connection in Control Hub, and when the Test Connection button is used in the Edit Connection window, the connection succeeds.  Only when it is used in an actual pipeline does it fail.CauseThis behavior occurs when two conditions are met:MySQL is configured to require SSL.  This condition can be met in one of three ways: The MySQL user is configured with the REQUIRE SSL option. The “requireSSL” property has been configured in the JDBC Additional Configuration Properties int he JDBC stage, with the value “true” “requireSSL=true” has been added to the JD

0
Douglas R
Fan
1 year ago
N
Fan
nik_shahFan
asked in Community Articles and Got a Question?
Writing value in Salesforce Master Detail Relationship field

I have a scenario where I need to populate the value of a parent object on its child record which is in a master detail relationship. For instance, I have a Custom Object “Customer Number” which is in Master Detail relationship with Leads. Instead of pass Salesforce ID, I want to pass the Lead.Name (as it is available in Streamsets) to establish the relationship of “Customer Number” record to its parent Lead object. What is the syntax that I need to use?

0
N
Fan
1 year ago

Badge winners

Show all badges
Terms & Conditions