Community Articles and Got a Question?

Below is my Api curl command but it shows error i followed the link  https://docs.streamsets.com/portal/platform-controlhub/controlhub/UserGuide/OrganizationSecurity/APICredentials_title.html#concept_vpm_p32_qqb.Shows ErrorError like:- 

ISSUE:While passing data-urlencode values via OAUTH 2 in HTTP client, We were getting error message-”HTTP_01 - Error fetching resource. HTTP-Status: 400 Reason: Bad Request”.The endpoint is microsoftonline oauth2. When we run below query in postman it works.curl --location --request GET 'https://login.microsoftonline.com/<xxx>/oauth2/v2.0/token \--header 'Content-Type: application/x-www-form-urlencoded' \--data-urlencode 'grant_type=client_credentials' \--data-urlencode 'client_id=<stripped>' \--data-urlencode 'client_secret=<stripped>' \--data-urlencode 'scope=https://xxx.com/.default'  but this is not working when we do this in HTTP client with below configuration.Configuration:Error Message:HTTP_01 HTTP_01 - Error fetching resource. HTTP-Status: 400 Reason: Bad Request  Resolution:Do add below property in HTTP  accept:*/*     We might be getting few error which can be fixed by following below process Error-"HTTP_00 - Cannot parse record. HTTP-Status:200 cause

I can use one origin only in pipeline and I am using jdbc query consumer and I want to call 3 store procedure  when I run or preview pipeline it shows me data of 1st store procedure only How to get data of all 3 store procedure ?

i want to schedule my job in monday to friday and the  time is 10 am to 6 pm IST and also every 2 minutes , how to schedule ,anyone suggest.

Issue: In SDC 5.5.0 and below, when configuring Snowflake stage, user is still forced to configure username and password even if key pair (private_key_file and private_key_file_password) are used in the Snowflake Connection Properties. This is due to the current implementation of the stage always requires username and password.  Solution: In SDC 5.6.0 and above, in Snowflake Connection Info tab, user can now choose to use Key Pair to connect to Snowflake without being prompted for User Credentials. Selecting Key Pair will allow the user to authenticate using Public Key and Private Key to make the connection to Snowflake.User can follow this Snowflake documentation https://docs.snowflake.com/en/user-guide/key-pair-auth to create a key pair and add Private Key path to pipeline configuration. To generate the keys, please use these following commands: For encrypted private key: openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -v1 PBE-SHA1-3DES -out rsa_key.p8 For non-encrypted priva

I am using jdbc query consumer and trying to connect to sql server but getting error of time out tcp connection port blocked by firewall but the port is open 

mysql error came  during writing how to fix error like:JDBC_23 - Can't coerce '[]' of type 'LIST' to column 'remarks' of JDBC type '-1' 

in the StreamSet Data Collector Kafka Multi Topic Consumer. I read that there is an option to configure the offset to earliest, latest or a particular timestamp I have a requirement to set the offset to start from a particular timestamp. Is there someone help me with how to achieve this or some examples?

as my origin is local director i want read multipile csv and process it by streamselector according to the file name =${record:value('/fileInfo/filename')=='submission_trackers_25523.csv'} but it shows error how can i handle this scenario.cd

but during write in mysql it shows error below like . 

Error: JDBC_00 - Cannot connect to specified database: java.sql.SQLException: Connection error: Check the host name and port number are correct, and that client and server belong to the same Denodo major version. Cannot trust the server to establish a SSL connection when connecting pipeline to Denodo using JDBC.Analysis: This is the issue with SSL certificate configuration. This might occur to due SSL misconfiguration or SDC not enabling SSL connection.Solution: You can first try to turn on SSL at the pipeline level by adding "?ssl=true" in the end of your connection string.For example, it could look like this:jdbc:vdb://<host>:<port>/database_name?ssl=trueIf the error still persists, please double check your SSL configuration to see if the certificate you imported to the truststore is valid, of if you have correctly pointed SDC to the right truststore.Below is the process you can use to extract Denodo cert and import to truststore:Use the following command to extract the c

 ProblemData Collector encountering OutOfMemory error: java.lang.OutOfMemoryError: Compressed class spaceThis error occurs when the capacity for classes and their metadata is exceeded. When the flag UseCompressedClassPointers is enabled, which it is by default, the capacity for class metadata is fixed at the amount set by CompressedClassSpaceSize. This value is 1G by default. SolutionExplicitly set the JVM option -XX:CompressedClassSpaceSize to a larger amount than 1G.Ex:export SDC_JAVA_OPTS="-XX:CompressedClassSpaceSize=2G ${SDC_JAVA_OPTS}" 

Suppose, I am getting 4 records in jython evaluator stage. I want to combine them and make them 2 records based on particular field.  {name: “Henry”,age: 12}{name : ”Henry”,emai: “Henry@gmail.com} The record should become like {name: “Henry”,age: 12,emai: “Henry@gmail.com} similarly if there 2 more records like that should combine separately. 

Extract updated data from mongocompass from specified time using streamsets pipelines,how can I do that any suggestions 

IssueUnable to use Hive Metastore destination in a Cloudera CDP 7.1.x environment with Kerberos.A stage library error occurs on the Hive Metastore destination stage. This does not occur with previous (6.3) version of the stage library.Environment Cloudera CDP 7.1.x Hive Metastore destination Stage library: CDP 7.1 Data Collector v4.x and 5.x, registered with Control Hub ‘Hadoop Impersonate current user’ option is enabled Root CauseWhen Data Collector is registered with Control Hub, authentication occurs with a different username in the format user@organization instead of the full username user@domain.com.In CDP 7.1.x, Cloudera made a change. Previously, a warning message would be printed in the log, but the stage library was not blocked, so the stage worked as expected. In CDP 7.1.x, an exception is thrown instead, which causes the stage library error.ResolutionChange the dpm.alias.name.enabled configuration option to true. More information on this setting can be found in the

QuestionHow to change the Garbage Collector settings for a Data Collector engine on StreamSets Platform?AnswerPlease refer to the Java Configuration Options topic in the latest Platform documentation for instructions. Use the settings listed in the Garbage Collector section.This procedure differs from the method used to change Java configuration on Legacy Data Collector. Do not modify files directly on the Data Collector engine in Platform; instead use the UI as explained in the documentation. NOTE: Setting the Garbage Collector to G1GC is only supported for Java 8. If you use Java 11 or 17, applying  these settings will result in the JVM being unable to start due to an invalid Java configuration option being specified.

IssueA pipeline fails due to a Kafka Stage with Kerberos authentication enabled is throwing the stage exception KAFKA_29 - Error fetching data from Kafka: org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: xxxxBackgroundThere are two general types of errors which can happen which prevent a user from connecting to a secured endpoint: Authentication and Authorization.Authentication errors - have to do with a credential failures, i.e. a Kerberos ticket is unable to be granted due to the wrong credentials being used, denying the principal from obtaining a service ticket. Authorization errors - occur after the user has made it passed authentication, but there is some additional Access Control Logic (ACL) which is preventing this user from accessing some resource (i.e. missing permissions to access a certain Kafka Consumer group).ResolutionThe GroupAuthorizationException signifies that the Kafka consumer has successfully authenticated and obtained a s

We are using Streamsets CDC stage for fetching Oracle CDC Data. We are received error like, Missing redo log files for time range and pipeline is not fetching any data for long.Question:What is the reason for receiving missing redo log error? What step need to take when we receive this kind of error?

Can I use StreamSets to ingest data to an on-premise Oracle DB? (Looking to move to the cloud)

I want to extract data from mongodb from the last modified date how can I do that, Is it possible to parameterize offset value in mongo origin 

As per the request we need to copy the files from sharepoint location using https client. but i am not able to find the proper data format to chose so that it will copy the zip file from the sharepoint location to Amazon s3 as it is. Kindly guide me to set the proper data format in order to copy the zip file from one location to other

Hi,I am unable to see the option for Field Delimiter and Record Delimiter in the ADLS Destination for the Multicharacter Delimiter Format in my Trailer Account.PFB the attached Snapshot --  Can you please look into this issue as the documentation states that it should be there. And can you please suggest if I want some of my output files to be in excel format then what should I do for it.

I have a StreamSets DC pipeline which I renamed and then checked in.The pipeline exists. But now When I run the same pipeline with a new name it is throwing the below errors:While running Pipeline: https://<url>:18443/rest/v1/pipeline/testRun__ 500 Internal Error.  While trying to run the job and capture snapshot. : Pipeline doesn’t exist or might have been stopped.Does anyone know why am I encountering this error?

I am using column type convertor  below configuration?                                                          but it not converted correct form. output:- 

I have created a simple transformer pipeline to run on an EMR cluster. In Cluster configuration, I have used the necessary configuration details.I am facing the following issue: