Skip to main content

Community Articles and Got a Question?

Can't find what you're looking for? Ask it here or check out the Community articles

  • 1,033 Topics
  • 2,323 Replies
1033 Topics
Kartik Medavarapu
Roadie
Kartik MedavarapuRoadie
asked in Community Articles and Got a Question?
DataOps Platform - REST API Origin - How to invoke pipeline?

Hi All,I created a microservice pipeline with REST API Origin using the DataOps platform as per video: https://www.youtube.com/watch?v=wIZWMV1bMl4Now, I am unable to invoke the pipeline using 3rd party tools like Postman. I have chosen default settings in SDC and headers as (X-SDC-APPLICATION-ID:sdc_microservice & Content-Type:application/json) yet I’m facing issue as “Error: connect ETIMEDOUT xx.xx.xx.xx:8000”.However, when I use the same URL in a curl command where SDC container is running, I see, 200 OK response. curl -i -X GET http://xx.xx.xx.xx:8000/rest/v1/user --header "X-SDC-APPLICATION-ID:sdc_microservice"There is no connectivity issue. So, can anyone tell me why I am unable to use Postman tool or if I am missing anything here?Below are the images showing the same: 

16712
AkshayJadhav
StreamSets Employee
1 year ago
S
Discovered Fame
saikat_chatterjeeDiscovered Fame
asked in Community Articles and Got a Question?
Facing error in On-prem setup of sch

I was trying to do the on-prem setup of SCH in which I have completed till configuring the control hub, relational databases, system data collector etc.Now when I am going to create the required tables for each databases using “sudo dev/01-initdb.sh” command the tables are getting created for the first database i.e. “Security” and it is throwing error in the next database i.e. “PipelineStore”From the marked red arrow the error is staring.Can someone help me regarding this?

494
S
Discovered Fame
1 year ago
Pradeep
StreamSets Employee
PradeepStreamSets Employee
posted in Community Articles and Got a Question?
Understanding and Resolving the NoClassDefFoundError: Common Causes and Solutions

The NoClassDefFoundError is a runtime error in Java that occurs if the Java Virtual Machine (JVM) or a ClassLoader instance attempts to load the definition of a class that could not be found. The class definition exists at compile-time but is not available at runtime.The definition of the class is attempted to be loaded as part of a normal method call or creating an instance of the class using the new expression and no definition of the class could be found. Therefore, it can occur during the linking or loading of the unavailable class.Common causes of the class definition being unavailable at runtime and solutions to try:Missing JAR file: First, identify the JAR file that contains the class for which you are receiving the NoClassDefFoundError. Once identified, verify if the JAR file exists in the installation directory structure of the StreamSets engine. Alternatively, check if it is located in other paths referenced by the data plane engine (e.g., the USER_LIBRARIES_DIR variable, whi

8600
Pradeep
StreamSets Employee
1 year ago
paul
StreamSets Employee
paulStreamSets Employee
posted in Community Articles and Got a Question?
Details on SDC Metrics

QuestionHow are the SDC Metrics collected and what do they represent?AnswerThe graphs on the SDC Metrics page in StreamSets Data Collector and the Metrics tab of the Execution Engine page in Control Hub are system-level metrics collected from standard Java core libraries. They should roughly correspond with metrics reported by other system-level tools, such as top and uptime at the command line, as well as external monitoring tools that show system-level metrics. Note that the numbers reported by different tools won’t match each other exactly due to differences in reporting intervals and other factors, but there should be a correlation.

250
paul
StreamSets Employee
1 year ago
xagavin
Fan
xagavinFan
asked in Community Articles and Got a Question?
​Use Oracle CDC Client the record value warped UNISTR function

​I used StreamSets Data Collector 5.4.0, Oracle character set is AL32UTF8, In preview mode, I see that the record value is wrapped in a layer of UNISTR functions, convert to unicodelike this :Oracle CDC Client config :is it something wrong with my JDBC config or how to set the character set ? thanks

876
xagavin
Fan
1 year ago
southwr62
Fan
southwr62Fan
asked in Community Articles and Got a Question?
How to create and use a variable

Hi, I am new to Streamsets, started using it today.I have a task to create some entries in a big query table, that represent the number of records in returned by an API call.I can do this using power automate, without issue, but power automate doesn’t support posting the BigQuery.roughly speaking the process I have in power automate is….Http get  array is returned get length of the array post the result to DB table I need to use Streamsets and BQ as these have been specified as a requirement.thanks in advance

1142
southwr62
Fan
1 year ago
K
Fan
kushal_upadhyayFan
asked in Community Articles and Got a Question?
Masking

Need to mask the fields using fixed length,custom and regular expression….Suppose I have given the field_to_mask parameter in the pipeline, how can I mask it ...For e.g in order to do the masking of fixed length the field to mask string is like this in the picture but its giving syntax error. Can you please provide and example how to do the masking in fixed,custom and regular expressionCan you please provide the examples of custom and regular expression

382
Bikram
Headliner
1 year ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Community Articles and Got a Question?
How to load the data to the elasticsearch destination?

In this article, our focus will be on utilizing the Elasticsearch 7.9.0 library to load data from a Kafka topic into Elasticsearch.We are going to use following environment:Kafka Consumer Origin [Kafka 2.7] Elasticsearch Destination [Elasticsearch Library 7.9.0]Currently, the source Kafka topic "elastic" contains approximately 5327 records. Now, let's delve into the details of our pipeline setup:Stage1: Consumer will fetch records from topic with the rate of 1000 records per batch. Stage2: We are going to remove the some unnecessary field from each record. Stage3: Credit card details will be masked and only last 4 digit will be visible.  Stage4: Upload the data to Elastic with index customer_data.Data Collector Pipeline flow.To run this pipeline, We need to setup the Elasticsearch deployment and get the username password + endpoint details from the ElasticSearch » Deployment section.Elasticsearch Deployment details. After successfully validating the pipeline, we proceeded to start the

510
AkshayJadhav
StreamSets Employee
1 year ago
molo
Fan
moloFan
asked in Community Articles and Got a Question?
Core SDC Tarball initialization failed

Local deployment of, error reported during startup

181
AkshayJadhav
StreamSets Employee
1 year ago
M
Fan
meowwFan
asked in Community Articles and Got a Question?
Error for Kafka Connection using aws secret manager as credential store

if i create a kafka connection using simple string then it works.. but if i create connection with these pramaters..${str:concat(str:concat(str:concat("org.apache.kafka.common.security.scram.ScramLoginModule required username='", credential:get("aws", "all@myorg.org", "MyStreamsetsSecret&kafka_username")), str:concat("' password='", credential:get("aws", "all@myorg.org", "MyStreamsetsSecret&kafka_password"))), "';")} then i am getting an below error if i test a connection  KAFKA_67 - Error connecting to Kafka Brokers 'org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient'

552
M
Fan
1 year ago
mblahay
Discovered Fame
mblahayDiscovered Fame
asked in Community Articles and Got a Question?
Alternate Git Repo Integration

When will StreamSets introduce the ability to use Git repos such as Github or GitLab for version control?

4097
saleempothiwala
Headliner
1 year ago
jeff
Fan
jeffFan
asked in Community Articles and Got a Question?
JDBC connection error with Streamsets DC Core Hour deployed from Azure Marketplace

Hi there,I just deployed a Streamsets Data Collector Core Hour 1.0 from Azure Marketplace for testing purpose, to stream MySQL binlog to storage.The MySQL binlog origin is throwing error : No appropriate protocol (protocol is disabled or cipher suites are inappropriate) .I have tried all the available options to troubleshoot this issue but it is still failing on the connection.MySQL server: v 8.0 Azure Database for MySQL Flexible ServerMySQL driver : mysql-connector-j-8.0.33.jarJDBC connection String : jdbc:mysql://hostname.mysql.database.azure.com:3306?useSsl=true&enabledTLSProtocols=TLSv1.2 I also tried to add tlsCiphersuites with one of the supported cipher suites that is available in this list https://docs.streamsets.com/portal/platform-datacollector/latest/datacollector/UserGuide/Pipeline_Configuration/SSL-TLS.html#concept_cwx_dyf_5z and the MySQL configuration parameter, but it does not work. Just to add that the mysql client command line can connect to the server from the St

200
jeff
Fan
1 year ago
blakeorsak
Fan
blakeorsakFan
posted in Community Articles and Got a Question?
JDBC Producer with Multi-Row Operations Parse Error (JDBC_14)

Issue:  When using a JDBC Producer destination with Multi-Row Operation enabled, the following error is thrown:  JDBC_14 - Error processing batch: [SQLState:S0001][ErrorCode:103010] Parse error at line: 1, column: 58: Incorrect syntax near ','. SQLState: S0001 Error Code: 103010 Message: Parse error at line: 1, column: 58: Incorrect syntax near ','. Solution:  This error is the result of certain databases not supporting the underlying SQL statements made by the JDBC Producer when this setting is enabled.  For example, for Multi-Row inserts, the JDBC Producer uses the following SQL statement: INSERT INTO <table name> (<col1>, <col2>, <col3>)      VALUES (<record1 field1>,<record1 field2>,<record1 field3>),      (<r2 f1>,<r2 f2>,<r2 f3>), (<r3 f1>,<r3 f2>,<r3 f3>),...; If the database does not support such statements, it will result  in the error message shown above. It is recommended to verify support for thes

1320
blakeorsak
Fan
1 year ago
R
Roadie
Rachit_2022Roadie
asked in Community Articles and Got a Question?
Orchestration Pipeline Transformer

Hi,I am new to Streamsets. I have a situation where i have created 3 pipelines on Transformer (Each pipeline runs 3-4 SQLs)  has Business Date as a parameter (Current Date-1).Use case - I need to trigger the first pipeline and when its completed then the second pipeline should run and when second pipeline is completed then third pipeline should be running. I have tried creating an orchestration pipeline on Data collector using start job, created job templates for each pipeline by watching some sample video and reading the documents but unable to achieve the above use case in the orchestration pipeline. Is there any other way to achieve this.Can anyone please help and advise. I would be obliged for any help received.

826
R
Roadie
1 year ago
R
Roadie
Rachit_2022Roadie
asked in Community Articles and Got a Question?
Referring a variable in Spark SQL Stage

Use Case - We have to pass dynamic date in Spark SQL Query stage in Streamsets,We have got an output as a dynamic date using pyspark code, now we need to use output in Spark SQL stages within SQL scripts. How to refer this date output variable in SQL scripts.Any help and advise would be great.

343
Bikram
Headliner
1 year ago
S
Discovered Fame
saikat_chatterjeeDiscovered Fame
asked in Community Articles and Got a Question?
pip3 install

I am trying to run the following command:“pip3 install streamsets~=5.0” in my VM to activate the sdk for python without using the activation key as shown in the documentation but unfortunately I am receiving the following error: WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])"))': /simple/streamsets/WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])"))': /simple/streamsets/WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])"))': /

1289
K
Fan
1 year ago
Mihir Kale
Roadie
Mihir KaleRoadie
asked in Community Articles and Got a Question?
Error while creating sdc instance in Python SDK

sch = ControlHub(API_CREDENTIAL_ID, API_TOKEN)sdc=DataCollector(server_url='http://023aee033abf:18630',control_hub=sch)sdc instance creation is taking lot of time to execute, when I force stop run, it shows the following err,-------------------------------------------------------------------------------------Traceback (most recent call last):  File "C:\Users\mihir_kale\PycharmProjects\Informatica_to_SS\sdc_instance.py", line 15, in <module>    sdc=DataCollector(server_url='http://023aee033abf:18630',control_hub=sch)        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "C:\Users\mihir_kale\PycharmProjects\Informatica_to_SS\venv\Lib\site-packages\streamsets\sdk\sdc.py", line 90, in __init__    self.api_client = sdc_api.ApiClient(server_url=current_server_url,                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "C:\Users\mihir_kale\PycharmProjects\Informatica_to_SS\venv\Lib\site-packages\streamsets\sdk\sdc_api.py", line 97, in __

929
Bikram
Headliner
1 year ago
K
Fan
kushal_upadhyayFan
asked in Community Articles and Got a Question?
Unable to fetch data using Kafka consumer random group

Unable to fetch data from kafka when trying kafka consumer to Local FS and getting follwoing error :iorg2.org] [runner:] [thread:ProductionPipelineRunnable-KAFKATOL__494a67a7-94ef-49e1-a41b-385c79551625__apiorg2.org-KAFKA_TO_LOCALFS] [stage:] INFO  AbstractRunner - Scheduling retry in '239996' milliseconds2023-05-11 23:21:31,730 [user:*admin@apiorg2.org] [pipeline:KAFKA_TO_LOCALFS/KAFKATOL__494a67a7-94ef-49e1-a41b-385c79551625__apiorg2.org] [runner:] [thread:ProductionPipelineRunnable-KAFKATOL__494a67a7-94ef-49e1-a41b-385c79551625__apiorg2.org-KAFKA_TO_LOCALFS] [stage:] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0800 - Can't start pipeline due 1 validation error(s). First one: KAFKA_41 - Could not get partition count for topic 'tst2' : com.streamsets.pipeline.api.StageException: KAFKA_41 - Could not get partition count for topic 'tst2' : org.apache.kafka.common.errors.Timeo

281
AkshayJadhav
StreamSets Employee
1 year ago
P
Discovered Fame
Priyanka MynepallyDiscovered Fame
asked in Community Articles and Got a Question?
curl command in shell

Hi Team  I need to post file1.json and file2.csv using curl.I tried this but no luckPlease help me  metadata=@/tmp/file1.json \files=@/tmp/file2.csv \curl -kiv --trace -w --location --request POST "<Location>" \--header "accept:application/json,application/hal+json, application/json, application/*+json, text/plain, application/octet-stream, application/xml, text/xml, application/x-www-form-urlencoded, application/*+xml, multipart/form-data, multipart/mixed, */*" \--header "Accept-Charset: UTF-8" \--header "Content-Type:application/json" \--form  metadata="$metadata" \--header "Accept-Charset: UTF-8" \--header "Content-Type:multipart/form-data,text/csv" \--form  files="$files"

2194
Bikram
Headliner
1 year ago
D
Fan
dhanraj_shindeFan
asked in Community Articles and Got a Question?
Edit an existing pipeline using python SDK

Hi TeamCan we edit a pipeline using python SDK. I tried creating new pipeline it worked. i tried to retrieving pipelines from control hub it worked!,but it only contain metadata of a pipeline like (name ,id commit id ,labels , version, ... etc ),I did not see any property related to stages and it’s configuration.  if it is possible, could you please share a code snippet of same.     

160
D
Fan
1 year ago
S
Roadie
shivrajRoadie
asked in Community Articles and Got a Question?
How to get stage configuration data in python SDk?

I have a pipeline object which was imported using pipeline Id. I need it’s stage configuration data.I have imported a pipeline but I wasn’t able find destination.How can I get a destination where file is getting stored after running pipeline. I want that path in runtime on python SDK.Please help me put with this concern asap.

220
S
Roadie
1 year ago
Saurabh Suman
Fan
Saurabh SumanFan
asked in Community Articles and Got a Question?
File generation on local

how can i generate a file from any source to local (on my computer) or a given unix server. 

211
R
StreamSets Employee
1 year ago
K
Fan
Karimulla ShaikFan
asked in Community Articles and Got a Question?
Read Emails.

Dears,Don’t we have SMTP connector to connect and get the email content like attachments/mail sub/body? 

301
saleempothiwala
Headliner
1 year ago
K
Fan
kanyewestnewmerch33Fan
asked in Community Articles and Got a Question?
How to handle auto increment columns while writing data into MySQL from MongoDB

I discovered that MongoDB lacks a 'id' field, although MySQL has one when transferring data between the two databases. I'm having trouble writing data into MySQL as a result. Regarding this situation, I need help managing the auto-increment column. Can anyone offer any advice or support?I'm also looking into the possibilities of handling this problem with the JDBC Tee processor. Please provide me some tips on how to use the JDBC Tee processor in this situation.

90
K
Fan
1 year ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in Community Articles and Got a Question?
cdc operation without use of mongodb oplog origin

If it is possible to achieve CDC (Change Data Capture) operations in MongoDB Atlas without using MongoDB Oplog, and only by utilizing MongoDB Atlas or other alternative methods, please provide the correct information or alternatives ,I am encountering an error when using MongoDB Oplog in my CDC operation in MongoDB Atlas. Although I have created a capped collection, the error still persists. Therefore, I am looking for alternative methods or processors to achieve CDC operations in MongoDB Atlas without relying on MongoDB Oplog.

210
lakshmi_narayanan_t
Discovered Fame
1 year ago

Badge winners

Show all badges
Terms & Conditions